No announcement yet.

C2750D4I Marvel 9230 SATA port problems

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • C2750D4I Marvel 9230 SATA port problems

    Hi All,
    I am in the process of migrating from 4x1 TB ZFS RAIDZ1 to 5x3TB ZFS RAIDZ2. It was nice to be able to have ALL the drives hooked up at the same time to do the setup and migration. However when I decided to test the new drives I ran into some problems. This is what I found.

    I am using Ubuntu 12.04.4 with the 3.5 kernel and ZFS on Linux. My test methodology was going to be to use dd to write to the drives and then compare the smart data to see how many re-mapped sectors happened. I had a problem with one of my drives dropping out but otherwise seemed ok. I was ready to send it back to NewEgg when I decided to continue testing. I switched to using badblocks instead of dd because it was able to write just as fast and did not cause kswap0 to use a lot of CPU like dd did and in addition it did verification of the writes (takes a looonnnnggg time with 3TB drives).

    In summary what I found is that I was not able to stably have more than 2 drives connected to the Marvell 9230 (it provides 4 of the SATA ports) under heavy load. These 4 ports are the first 4 white ones as you count from the edge of the board. The top 2 are connected to the Marvell 9172 and appear fine. My current setup that works has:

    4x1TB old drives, connected to 4 blue ports which are the SATA2 ports provided by the Avoton.
    1x250 SSD connected to one of the white SATA3 ports next to the blue ports provided by Avoton
    1x3TB connected to other white SATA3 from Avoton
    2x3TB connected to Marvell 9172 ports
    2x3TB connected to Marvell 9230 ports

    Every test I tried with more than 2 drives connected to the 9230 failed with entries in the kernel log and eventually the drive would get disconnected. I even shuffled drives around to make sure it was no the specific drive. It did not make a difference.

    Has anyone else experienced this? I am on BIOS 1.8.

    I am currently in communication with ASRock and have provided them all my test configurations and am very impressed with their responsiveness so far. Hopefully it is something they can fix with a BIOS update or who knows, I guess it could also be a Linux kernel issue. This is why I am putting this out there to see if other have had similar issues and could provide more info to ASRock.

    This is what my kern.log entries looked like:

    Jan 25 21:42:15 zfs kernel: [ 615.011152] ata9.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
    Jan 25 21:42:15 zfs kernel: [ 615.011329] ata9.00: failed command: SMART
    Jan 25 21:42:15 zfs kernel: [ 615.011462] ata9.00: cmd b0/d0:01:00:4f:c2/00:00:00:00:00/00 tag 0 pio 512 in
    Jan 25 21:42:15 zfs kernel: [ 615.011462] res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
    Jan 25 21:42:15 zfs kernel: [ 615.011908] ata9.00: status: { DRDY }
    Jan 25 21:42:15 zfs kernel: [ 615.012024] ata9: hard resetting link
    Jan 25 21:42:16 zfs kernel: [ 615.338638] ata9: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
    Jan 25 21:42:21 zfs kernel: [ 620.330962] ata9.00: qc timeout (cmd 0xec)
    Jan 25 21:42:21 zfs kernel: [ 620.338942] ata9.00: failed to IDENTIFY (I/O error, err_mask=0x4)
    Jan 25 21:42:21 zfs kernel: [ 620.338948] ata9.00: revalidation failed (errno=-5)
    Jan 25 21:42:21 zfs kernel: [ 620.339051] ata9: hard resetting link
    Jan 25 21:42:21 zfs kernel: [ 620.666451] ata9: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
    Jan 25 21:42:31 zfs kernel: [ 630.651103] ata9.00: qc timeout (cmd 0xec)
    Jan 25 21:42:31 zfs kernel: [ 630.659078] ata9.00: failed to IDENTIFY (I/O error, err_mask=0x4)
    Jan 25 21:42:31 zfs kernel: [ 630.659084] ata9.00: revalidation failed (errno=-5)
    Jan 25 21:42:31 zfs kernel: [ 630.659185] ata9: limiting SATA link speed to 3.0 Gbps
    Jan 25 21:42:31 zfs kernel: [ 630.659193] ata9: hard resetting link
    Jan 25 21:42:31 zfs kernel: [ 630.986591] ata9: SATA link up 6.0 Gbps (SStatus 133 SControl 320)
    Jan 25 21:43:01 zfs kernel: [ 660.940566] ata9.00: qc timeout (cmd 0xec)
    Jan 25 21:43:01 zfs kernel: [ 660.948541] ata9.00: failed to IDENTIFY (I/O error, err_mask=0x4)
    Jan 25 21:43:01 zfs kernel: [ 660.948546] ata9.00: revalidation failed (errno=-5)
    Jan 25 21:43:01 zfs kernel: [ 660.948642] ata9.00: disabled
    Jan 25 21:43:01 zfs kernel: [ 660.956535] ata9: hard resetting link
    Jan 25 21:43:02 zfs kernel: [ 661.276053] ata9: SATA link up 6.0 Gbps (SStatus 133 SControl 320)
    Jan 25 21:43:02 zfs kernel: [ 661.284034] ata9: EH complete
    Jan 25 21:43:02 zfs kernel: [ 661.284215] sd 8:0:0:0: [sdg] Unhandled error code
    Jan 25 21:43:02 zfs kernel: [ 661.284222] sd 8:0:0:0: [sdg]
    Jan 25 21:43:02 zfs kernel: [ 661.284225] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
    Jan 25 21:43:02 zfs kernel: [ 661.284231] sd 8:0:0:0: [sdg] CDB:
    Jan 25 21:43:02 zfs kernel: [ 661.284234] Write(10): 2a 00 00 1a 00 00 00 04 00 00
    Jan 25 21:43:02 zfs kernel: [ 661.284249] end_request: I/O error, dev sdg, sector 1703936
    Jan 25 21:43:02 zfs kernel: [ 661.284252] sd 8:0:0:0: [sdg] Unhandled error code
    Jan 25 21:43:02 zfs kernel: [ 661.284256] sd 8:0:0:0: [sdg]
    Jan 25 21:43:02 zfs kernel: [ 661.284257] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
    Jan 25 21:43:02 zfs kernel: [ 661.284259] sd 8:0:0:0: [sdg] CDB:
    Jan 25 21:43:02 zfs kernel: [ 661.284260] Write(10): 2a 00 00 1a 7c 00 00 04 00 00
    Jan 25 21:43:02 zfs kernel: [ 661.284269] end_request: I/O error, dev sdg, sector 1735680
    Jan 25 21:43:02 zfs kernel: [ 661.284288] sd 8:0:0:0: [sdg] Unhandled error code
    Jan 25 21:43:02 zfs kernel: [ 661.284291] sd 8:0:0:0: [sdg]
    Jan 25 21:43:02 zfs kernel: [ 661.284292] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
    Jan 25 21:43:02 zfs kernel: [ 661.284295] sd 8:0:0:0: [sdg] CDB:
    Jan 25 21:43:02 zfs kernel: [ 661.284296] Write(10): 2a 00 00 1a 80 00 00 04 00 00
    Jan 25 21:43:02 zfs kernel: [ 661.284306] end_request: I/O error, dev sdg, sector 1736704

  • #2
    Re: C2750D4I Marvel 9230 SATA port problems

    You mention trying different drives, but have you tried different cables? Other than that, I'd suggest trying a newer version of Ubuntu. Good luck.

    Comment


    • #3
      Re: C2750D4I Marvel 9230 SATA port problems

      I'm having the exact same issue as described here...
      Did you ever find a solution?

      I've tried 12.04 thru 12.04.4 and also 14.04 (all 64-bit versions).

      For some reason ASRock support has not replied to either of the messages I submitted to their support ticket system.

      Comment


      • #4
        Re: C2750D4I Marvel 9230 SATA port problems

        Do you know if your board has one or two 9230 chips? Some of the ASRock standard (non-server type) boards use two ASMedia SATA chips to provide four extra SATA ports. Not that the ASMedia chips are better, they aren't IMO, and only use one PCIe 2.0 lane per chip. I think your board has one 9230.

        IMO, two drives on one PCIe 2.0 lane like most of the Marvell 91xx SATA chips is to much for it to handle when pushing both drives at the same time. Not that that is an unreasonable thing to do, but just what happens.

        Although the Marvell 9230 SATA chip finally uses two PCIe 2.0 lanes, using more than two drives on it may stress it out and cause the errors you are seeing. The SATA driver you use with it may also make a difference.

        Comment


        • #5
          Re: C2750D4I Marvel 9230 SATA port problems

          Originally posted by parsec View Post
          Do you know if your board has one or two 9230 chips? Some of the ASRock standard (non-server type) boards use two ASMedia SATA chips to provide four extra SATA ports. Not that the ASMedia chips are better, they aren't IMO, and only use one PCIe 2.0 lane per chip. I think your board has one 9230.

          IMO, two drives on one PCIe 2.0 lane like most of the Marvell 91xx SATA chips is to much for it to handle when pushing both drives at the same time. Not that that is an unreasonable thing to do, but just what happens.

          Although the Marvell 9230 SATA chip finally uses two PCIe 2.0 lanes, using more than two drives on it may stress it out and cause the errors you are seeing. The SATA driver you use with it may also make a difference.
          The C2750D4I has 2 additional SATA chips:
          1) 88SE9172
          1) 88SE9320

          Hope that answers your question.

          Comment


          • #6
            Re: C2750D4I Marvel 9230 SATA port problems

            I googled and found this thread, and then I registered immediate.

            This is EXACTLY the same issue I have. Mine is Asrock 2550, which uses the same Marvell controller as 2750 motherboard.

            The error message is very similar to mine. When I was copying large files (TB), at first it was alright, then it broke down. I also did Snapraid sync job, but it kept failing.

            Interestingly, when I use Teracopy software to copy stuff, it broke, but with windows8 internal copy program it was all right. I blamed Teracopy at first, but obviously it is not completely its fault. Snapraid got broken then I guess it is the SATA controller issue. After I switch the cable to intel SATA, everything is fine so far.

            This is really a cool board and I don't want to give it up. Any way to fix?

            Oh, the system is Ubuntu 14.04 64-bit.

            I just hope more people can find this and then Asrock or Marvell or Ubuntu people can fix it.

            Comment


            • #7
              Re: C2750D4I Marvel 9230 SATA port problems

              Originally posted by littlelio View Post
              I googled and found this thread, and then I registered immediate.

              This is EXACTLY the same issue I have. Mine is Asrock 2550, which uses the same Marvell controller as 2750 motherboard.

              The error message is very similar to mine. When I was copying large files (TB), at first it was alright, then it broke down. I also did Snapraid sync job, but it kept failing.

              Interestingly, when I use Teracopy software to copy stuff, it broke, but with windows8 internal copy program it was all right. I blamed Teracopy at first, but obviously it is not completely its fault. Snapraid got broken then I guess it is the SATA controller issue. After I switch the cable to intel SATA, everything is fine so far.

              This is really a cool board and I don't want to give it up. Any way to fix?

              Oh, the system is Ubuntu 14.04 64-bit.

              I just hope more people can find this and then Asrock or Marvell or Ubuntu people can fix it.
              The technical staff from ASRock sent me an email late last week, but I haven't had an opportunity to try the fix yet (updated BIOS, baseband and 9230 firmware). I'll write back on here later this week once I've updated.

              Comment


              • #8
                Re: C2750D4I Marvel 9230 SATA port problems

                It must be a kernel or driver related issue. I have mine running for couple of weeks now using Centos ZFS with 5 of the 6 Marvel sata ports without issue. I do not have a need for the last port yet but that is also working when I tried it but only for a few days.

                Comment


                • #9
                  Re: C2750D4I Marvel 9230 SATA port problems

                  Thanks.

                  The error is not quite repeatable. Yesterday I tried two new WD Green 3T drives and it broke. But today I tried one very old WD Green 1T drive, so far so good.

                  R4D4R, it is great Asrock replied you. If you want I can even try my end. My server is new and it is still in test phase so I can play with it.

                  Comment


                  • #10
                    Re: C2750D4I Marvel 9230 SATA port problems

                    Originally posted by R4D4R View Post
                    The technical staff from ASRock sent me an email late last week, but I haven't had an opportunity to try the fix yet (updated BIOS, baseband and 9230 firmware). I'll write back on here later this week once I've updated.
                    Hi R4D4R, please keep me posted of your result. Thanks!

                    Comment


                    • #11
                      Re: C2750D4I Marvel 9230 SATA port problems

                      Any updates on this?

                      Comment


                      • #12
                        Re: C2750D4I Marvel 9230 SATA port problems

                        Latest BIOS P2.40 update and BMC firmware 0.17.0 are available ASRock Rack > C2750D4I . Users of C2750D4I ran into problem are encouraged to try to update the BIOS and BMC altogether.

                        Click image for larger version

Name:	c2750update.PNG
Views:	2
Size:	16.0 KB
ID:	754493

                        Bill
                        ASRock Rack
                        [email protected]

                        Comment


                        • #13
                          Re: C2750D4I Marvel 9230 SATA port problems

                          Hi all, long time since my original post and I have received a couple PMs about this recently so I thought I would update my situation with what I told them.

                          I was never able to have more than 2 drives on the Marvel 9230 (first 4 white connectors from the edge). So as long as you don't need more than 10 drives you should be ok. I am only using 7 (6 HDD + 1 SSD) and the system has been rock solid since I set it up end at the beginning of the year, though I did update it to Ubuntu 14.04.

                          I did get contacted by ASRock support several months ago about an updated firmware but never tried it because I did not want to take my system offline for an extended period and also because I did not have spare drives to do the destructive tests I was doing before.

                          Hope this helps others.
                          -Carlos
                          Last edited by create; 11-12-2014, 06:20 PM. Reason: formatting

                          Comment


                          • #14
                            Re: C2750D4I Marvel 9230 SATA port problems

                            So I am back!

                            I haven't done too much with my C2550D4I, until today. I copied large amount of data to the server, and was trying to do snapraid sync. However after several attempts, the program said it could not read file. It differs among the attempts. But these are new hard drives! Am I that lucky? I tested the harddrive, but it seems fine.

                            Then I realized something: I bought a new server case last month, and moved all the hardware into the new case. And I plugged the HHD cable into different slots! Now I remember this thread, that the 9230 chip is buggy! Now I have re-plug in the HDD cable into other sata ports - NOT the 9230! so far so good!

                            I has been some time, but unfortunately there is no new firmware for C2550! However, there is new firmware for C2570. That's unfair... I think they are very similar?

                            Could I ask if any Asrock guy could update it? I am willing to try it out! Thanks!

                            Comment


                            • #15
                              Re: C2750D4I Marvel 9230 SATA port problems

                              So this time I was able to find something and kind of fixed it!

                              I have apologize that I said Asrock 'did nothing' about 9230 issue. When I was just checking the C2570 webpage, I saw some texts like "To disable Marvell SE 9230 HW Raid, please click here", and here is the link ASRock Rack > Support where also provides a package for 9230 on C2550. And here is the description for the fix:

                              Since we found Marvell 9230 Raid Mode get some problem at some OS, we already highlighted this issue to Marvell.
                              In the short term solution, can disable Marvell 9230 HW Raid function to make Marvell 9230 port more stable, please follow below procedure to flash Marvell 9230 FW to disable HW Raid function.

                              So I boot the server into DOS, simply run the firmware updater to disable the raid function. Then connected all the disk back to 9230 ports, run the same snapraid sync command, and it went through without any problem.

                              I recon this is a good news! More tests may be needed, but it look good! Any updates I will let you guys now. Thanks.

                              Comment

                              Working...
                              X