C2750D4I - Purchased from NewEgg purchase a few months ago and unboxed recently and keeps locking up/crashing

[Retteren]

Re: C2750D4I - Purchased from NewEgg purchase a few months ago and unboxed recently and keeps locking up/crashing

See... terms like this just give me hope on mine. Thank you. :)

At your service :)
Build details/talks are here: https://www.reddit.com/r/zfs/comment...s_build_20tib/

I know I know... fam/friend tech support sux... doing it for them keeps me entertained at the same time.

Ah, the joys of involuntary and unpaid IT tech support to the extended family. I can live with the Windows 7 and Linux stuff, but Win8 and Macs just baffle me.

[arcjn97]

Re: C2750D4I - Purchased from NewEgg purchase a few months ago and unboxed recently and keeps locking up/crashing

Just received my board back from RMA yesterday. Got it all setup last night with mixed RAM sticks (1x 4GB and 1x 8GB). 8 storage devices connected on it too:1x SSD on the first Intel port with the rest being HDD of varying sizes. So far it's been running fine for over 2 hours and 30 mins - no load yet and mostly idle. Fingers crossed hoping that it will keep going until I finish work so I can do some real tests.

I am already thankful to the AsRock guys given that this board never got this far with everything connected on it.

Some stats:
From power button press to monitor wakeup: 53.01s
BIOS finished to Ubuntu boot up: 7.3s
Ubuntu desktop finished loading: 15.36s
Time on clock when desktop loaded: 0903
Time now (still running): 1140

Will get 11x 8TB HDD for this once I can confirm that this is solid.

[arcjn97]

Re: C2750D4I - Purchased from NewEgg purchase a few months ago and unboxed recently and keeps locking up/crashing

First crash... looks like a samba related lock up by the looks of the logs. Time now is 16:44. This is definitely something different now. I'll do a clean OS re-install tonight and try a stress test all day tomorrow.

[Retteren]

Re: C2750D4I - Purchased from NewEgg purchase a few months ago and unboxed recently and keeps locking up/crashing

Great to hear that the new board has resolved the big issues. As for the crash, you might want to take a look at the ipmi logs as well, to see what happened.

Clean OS re-install and stress testing sounds like a good plan.
I'd suggest to run "badblocks -w -c <large number>" on all the HDD's in parallel to stress-test the SATA subsystem (controller + cables) and burning in the HDDs.

Beware that running badblocks with the default 4 patterns (0xaa, 0x55, 0xff, 0x00) will literally take days. WD Red's can push about 170MB/sec sustained, and with your 8TB disks, you're looking at ~105 hours for a single pass through the 4 patterns. In the end, I opted for only doing one pattern.

[arcjn97]

Re: C2750D4I - Purchased from NewEgg purchase a few months ago and unboxed recently and keeps locking up/crashing

Well... stress testing with Prime95 as detailed here: How to test your PC for the Skylake bug | PCWorld hangs it up within the 30 minute mark all the time.

So far running badblocks with one pattern runs it for an hour. I have observed the same when running chkdsk <drive_letter> /f /v /r /b in Windows. The symptom is different compared to the CPU testing though. CPU stress test hangs it up. The tests on the disks or leaving it idle (left it idle last night) gives is the random power off. The longest uptime that I have observed so far is over 8 hours - which I am sort of content with but still in the state of searching for a config that will make this even more stable. I am trialing out the disable on the C-states and putting a cap on max CPU utilization to 80%. If I can get a solid uptime on these settings... I am a happy camper.

Wish me luck!

[Retteren]

Re: C2750D4I - Purchased from NewEgg purchase a few months ago and unboxed recently and keeps locking up/crashing

Well... stress testing with Prime95 as detailed here: How to test your PC for the Skylake bug | PCWorld hangs it up within the 30 minute mark all the time.

So far running badblocks with one pattern runs it for an hour. I have observed the same when running chkdsk <drive_letter> /f /v /r /b in Windows. The symptom is different compared to the CPU testing though. CPU stress test hangs it up. The tests on the disks or leaving it idle (left it idle last night) gives is the random power off.

Wish me luck!

I haven't tried stress testing the CPU on my C2750D4I board and since it's a 'production' system now, I'm unwilling to take it down for testing now.

Random power-offs doesn't sound good. They shouldn't happen. I've read about very odd problems that were caused by marginal PSUs. Did I understand post #8 correct that you have already tried with another 1000W PSU? Mine is a Corsair RM450 and I have had zero problems with it (but then again, it's not loaded very hard at all).

[arcjn97]

Re: C2750D4I - Purchased from NewEgg purchase a few months ago and unboxed recently and keeps locking up/crashing

After going through the RMA process I still winded up with the same issue. I have asked the guys in Yuan Pan and Asrock Rack what was done on the board given that I was told that it has been repaired. The board arrived on 04/04/16 and after all the failed testing, I asked them the question on the 7th which unfortunately is yet to be answered.

It looks like there was just a new grey thermal paste that was added on under the heat sink. I figured that they too have the same train of thought about the lock ups - it's typically caused by the overheating protection of the board. Most boards have this even laptops. This is the reason why old laptops with blocked heat sinks tend to have the same symptoms - they lock up or just power down.

I then look into replacing this thermal compound with the Cooler Master V1 thermal paste. This is the paste that I put into it initially. I was surprised that the heat sink did not have any to begin with when I got the box new. I then tried to run the board using 1x 12cm system fan hooked into the CPU_fan connection into the board. It booted up and stayed on for 1 hour. What I noticed is that the lock up this time around happens when the CPU goes into it's minimum state. Via the IPMI, I then noticed that the 12v power tend to oscillate. This is evident in the IPMI event log too. The times that it locks up coincides with the time stamps when it either becomes low or high and is unrecoverable in value. This Corsair HX1000W is a dual rail with the 2x 12 volt rail rated at 40amps. This only happens when I plug a fan in the CPU_fan connector on the board too.

It appears that I have been dealing with multiple root causes here. There's the heat... but if I resolve that by attaching a fan I get the voltage oscillation problem. Similar symptoms (power downs and lock ups) but different underlying causes.

I then tried a very old no name 1000W PSU (from memory, this was a $40 PSU compared to the HX1000W which is a $350 one) but this time I hooked up the fans into the molex connectors. 29 hours later it is still running strong! The first 2 hours had Prime 95 stress testing + chkdsk (/f /v /r /b options) running through a 60tb composed of 10x stripe raid of 6tb disks (still going this one) - this uses the entire 32GB of ram. I have connected 2 of those 12cm fans so the airflow doesn't just cools the CPU heat sink. It cools the entire board. Air flows from the front of the case, going out to the back.

The monitoring is being done via the IPMI. I have temporarily hooked up LAN port 1 to the IPMI, bridged the 2 LAN ports, and have the LAN port 2 connected to my local network. This way, I can RDP into it using my phone to check if any of the testing is still going and still see the stats remotely via IPMI. I have ran out of port in my local network that's why I did this. Otherwise, I would have activated the teaming function and plug all 3 into the network.

Thanks to all the guys in this very good forum that helped me keep my faith and composure.
William - you, my friend, are a legend! Most responsive motherboard support guy that I ever interacted with. Your responsiveness also add in to the bit that kept this going. Thank you!

I really hope this brings this to a close - well... I ain't shutting it down until the next week. If there's no updates then, it's confirmed resolved.

[arcjn97]

Re: C2750D4I - Purchased from NewEgg purchase a few months ago and unboxed recently and keeps locking up/crashing

Just want to add... I am never shopping at Newegg ever again. They only have 15-days cover to do any RMA.
I'll stop there because I have nothing nice to say.

[Retteren]

Re: C2750D4I - Purchased from NewEgg purchase a few months ago and unboxed recently and keeps locking up/crashing

arckn97, great to hear that your system is stable.

However, I'm still a bit concerned that you are seeing heat and PSU related issues.

I haven't done anything to my board wrt thermal paste - the system just runs. As a stress-test, I started 16 identical "bzip2" compression processes, trying to compress a 6.4GB video file (ie: it's not compressible). This pins all 8 CPUs to 100%. I later added a ZFS scrub to activate the 6 disks as well. I'm not exercising the memory in particular, simply because I couldn't come up program to use.

The system has now run ~2 hours with this load.
The CPU temp has stabilized at 68C and the mobo temp at 50C. I have 2 x 90mm fans placed in front of the disks, which run at 1400-1500rpm. SMART data tells me that the disks have a temperature of 24-29C. In the rear of the chassis I have a 120mm fan, which idles at 1100 rpm.
All three fans are 4-pin PWM controlled fans, plugged into the REAR_FAN1 (90mm), READ_FAN2 (90mm) and FRNT_FAN1 (120mm) headers.
The voltage rails doesn't seem to have any problems or oscillations. The system is just rock-solid, bullet proof.

In my book, you should be seeing the same solidness.

Adding the thermal paste should just help. As long as there is just a minimum of airflow, you shouldn't be seeing overheating issues, IMHO.

That only leaves me with the PSUs. Generally, PSUs are designed for optimum efficiency with a load close to their maximum. In fact, putting too little load up a voltage rail could be a problem, if the PSU isn't designed to correctly handle a very light load. Some PSUs will simply shut off if they aren't loaded, as a preventive measure (otherwise the voltage could overshoot dangerously).

My system draws about 50W from the wall, and your system likely to only draw slightly more - maybe 75W, given the extra disks. The only time that the PSU could see a large current draw would be at spin-up of the disks, unless the spin-ups are staggered in time.

This is leading be back to wonder if using a lower wattage PSU could actually be beneficial for your system stability.

I've added a screenshot of my IPMI temp data for your reference.

[arcjn97]

Re: C2750D4I - Purchased from NewEgg purchase a few months ago and unboxed recently and keeps locking up/crashing

Thank you for sending your stats! This indeed confirms what I am seeing is normal.

For your reference:

This is my settings for Prime 95 - to push all 8 cores to 100%:


This is the check disk output - to push the memory usage as close to 100% and disk controllers simultaneous usage:


IPMI readings during test:


IPMI readings after tests:


The whole thing draws 110 watts from the wall with all the components being used. On idle, this drops to 32 watts - which is awesome!

My main issue was the oscillating 12v rail (which goes +/-0.5v) on that multi-rail PSU. This is because even when I added fans to cool it off to solve the heating problem, the PSU's behaviour when a very low voltage board is hooked up is not acceptable. They are both rated at 1000 watts. The one that is working right now also has a single rail design and only varies +/- 0.1v at the very most on the said 12v rail. I must say this is the deepest I have dug in ever on building a PC. Any other time is like playing legos.

Next step is to go all green with it. I have a several solar panel up my roof. I want to hook this up with my setup of 5kw pure sine inverter with deep cycle battery banks. So during the day, it runs for free + batteries charging (also for free) and runs through batteries at night (also for free).

[Retteren]

Re: C2750D4I - Purchased from NewEgg purchase a few months ago and unboxed recently and keeps locking up/crashing

Yeah, a pity that you've had to work so hard to make the system work. The biggest issue I've had when building PCs have been a WD Green 1.5TB drive which started throwing SMART errors within a couple of months. Fortunately, it was part of a RAID1 mirror, so I could just RMA the disk and install the replacement.

Sounds nice with the solar panels :)

[profJim]

Re: C2750D4I - Purchased from NewEgg purchase a few months ago and unboxed recently and keeps locking up/crashing

How old is your HX1000 power supply?
The voltage drop of 0.5 volts on the 12 volt rails does not sound good with low system loads.

With an older 1000 watt power supply running at less that 20% of it's rated capacity, the efficiency will be below 80%.
Optimal psu efficiency is usually when it's running between 40% - 80% of its rated capacity.

You've done three tests with different power supplies:

1. Aerocool VX400-V3X Black
2. Corsair HX1000
3. No name inexpensive 1000 watt

In tests #1 and #2 it sounds like your cooling fans were connected to the motherboard's fan headers.

Is this correct?

What bios setting was used for the fan header speed settings?

Were the fans running at full speed at all times during test #1 and #2?

In test #3 the fans were connected directly to the power supply's 12 volt connector instead of the fan headers so the fans should be running at full speed all of the time.

I suspect that there might be a problem with the fan headers' speed control.
Running a test with the no name 1000 watt power supply with the fans connected to the motherboard's fan headers should help determine if there is a problem.

You should rerun your tests where HWiNFO is monitoring temperatures, voltages and fan speeds every 2 seconds to see detailed information throughout the test. Even if your system locks up or you have a BSOD, the CSV file that HWiNFO creates will still be usable. I reset HWiNFO's sensor readings immediately before I start the monitoring and running the stress tests.




Corsair HX1000 Power Supply Review, May 2008 (jonnyGURU)

[arcjn97]

Re: C2750D4I - Purchased from NewEgg purchase a few months ago and unboxed recently and keeps locking up/crashing

How old is your HX1000 power supply?

Really old but the single rail 1000w PSU is older by at least a year.

With an older 1000 watt power supply running at less that 20% of it's rated capacity, the efficiency will be below 80%.
Optimal psu efficiency is usually when it's running between 40% - 80% of its rated capacity.

True although after the power consumption reading (when I removed all the HDD and leaving only the boot SSD in with 32GB of RAM) that I have done and getting 32 watts on idle and 37 watts on full CPU + RAM usage (done with P95) - I was a very happy camper. I've lost count on the days that the system is running problem free. Got the 8TB HDDs connected on it and all running for free - This is now connected to a 5kw pure sine inverter + battery bank + solar.

The tests of having the fans connected to the board and off the board were done on all 3 PSU. The issue is primarily on that HX100W PSU - due to the power oscillation "hidden feature". The heating problem is present on all the PSU (if I don't put any fan on) especially during the stress testings.

In the end, I kept the fans off the mobo. I don't care if they are on 100% duty all the time... it runs for free.