SuperServer SYS-221BT-HNC9R - Server Supermicro - Free user manual and instructions

USER MANUAL SuperServer SYS-221BT-HNC9R Supermicro

Front view of a black server rack with multiple drive bays and orange buttons (no visible text or labels)

USER'S MANUAL

Revision 1.0d

The information in this User's Manual has been carefully reviewed and is believed to be accurate. The vendor assumes no responsibility for any inaccuracies that may be contained in this document, and makes no commitment to update or to keep current the information in this manual, or to notify any person or organization of the updates. Please Note: For the most up-to-date version of this manual, please see our website at www.supermicro.com.

Super Micro Computer, Inc. ("Supermicro") reserves the right to make changes to the product described in this manual at any time and without notice. This product, including software and documentation, is the property of Supermicro and/or its licensors, and is supplied only under a license. Any use or reproduction of this product is not allowed, except as expressly permitted by the terms of said license.

IN NO EVENT WILL Super Micro Computer, Inc. BE LIABLE FOR DIRECT, INDIRECT, SPECIAL, INCIDENTAL, SPECULATIVE OR CONSEQUENTIAL DAMAGES ARISING FROM THE USE OR INABILITY TO USE THIS PRODUCT OR DOCUMENTATION, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. IN PARTICULAR, SUPER MICRO COMPUTER, INC. SHALL NOT HAVE LIABILITY FOR ANY HARDWARE, SOFTWARE, OR DATA STORED OR USED WITH THE PRODUCT, INCLUDING THE COSTS OF REPAIRING, REPLACING, INTEGRATING, INSTALLING OR RECOVERING SUCH HARDWARE, SOFTWARE, OR DATA.

Any disputes arising between manufacturer and customer shall be governed by the laws of Santa Clara County in the State of California, USA. The State of California, County of Santa Clara shall be the exclusive venue for the resolution of any such disputes. Supermicro's total liability for all claims will not exceed the price paid for the hardware product.

FCC Statement: This equipment has been tested and found to comply with the limits for a Class A or Class B digital device pursuant to Part 15 of the FCC Rules. These limits are designed to provide reasonable protection against harmful interference when the equipment is operated in industrial environment for Class A device or in residential environment for Class B device. This equipment generates, uses, and can radiate radio frequency energy and, if not installed and used in accordance with the manufacturer's instruction manual, may cause harmful interference with radio communications. Operation of this equipment in a residential area is likely to cause harmful interference, in which case you will be required to correct the interference at your own expense.

California Best Management Practices Regulations for Perchlorate Materials: This Perchlorate warning applies only to products containing CR (Manganese Dioxide) Lithium coin cells. "Perchlorate Material-special handling may apply. See www.dtsc.ca.gov/hazardouswaste/perchlorate".

WARNING: This product can expose you to chemicals including lead, known to the State of California to cause cancer and birth defects or other reproductive harm. For more information, go to www.P65Warnings.ca.gov.

The products sold by Supermicro are not intended for and will not be used in life support systems, medical equipment, nuclear facilities or systems, aircraft, aircraft devices, aircraft/emergency communication devices or other critical systems whose failure to perform be reasonably expected to result in significant injury or loss of life or catastrophic property damage. Accordingly, Supermicro disclaims any and all liability, and should buyer use or sell such products for use in such ultra-hazardous applications, it does so entirely at its own risk. Furthermore, buyer agrees to fully indemnify, defend and hold Supermicro harmless for and against any and all claims, demands, actions, litigation, and proceedings of any kind arising out of or related to such ultra-hazardous use or sale.

Manual Revision 1.0d

Release Date: July 10, 2023

Unless you request and receive written permission from Super Micro Computer, Inc., you may not copy any part of this document. Information in this document is subject to change without notice. Other products and companies referred to herein are trademarks or registered trademarks of their respective companies or mark holders.

Copyright © 2023 by Super Micro Computer, Inc.

All rights reserved.

Printed in the United States of America

Preface

About this Manual

This manual is written for professional system integrators and PC technicians. It provides information for the installation and use of the server. Installation and maintenance should be performed by experienced technicians only.

Please refer to the SYS-221BT-H Series server specifications page on our website for updates on supported memory, processors and operating systems (http://www.supermicro.com).

Notes

For your system to work properly, please follow the links below to download all necessary drivers/utilities and the user's manual for your server.

• Supermicro product manuals: http://www.supermicro.com/support/manuals/
• Product drivers and utilities: https://www.supermicro.com/wdl
• Product safety info: http://www.supermicro.com/about/policies/safety_information.cfm

If you have any questions, please contact our support team at:

[e-mail]

This manual may be periodically updated without notice. Please check the Supermicro website for possible updates to the manual revision level.

Secure Data Deletion

A secure data deletion tool designed to fully erase all data from storage devices can be found on our website: https://www.supermicro.com/about/policies/disclaimer.cfm?url=/wdl/utility/Lot9_Secure_Data_Deletion_Utility/

Warnings

Special attention should be given to the following symbols used in this manual.

Warning! Indicates important information given to prevent equipment/property damage or personal injury.

Warning! Indicates high voltage may be encountered when performing a procedure.

Contents

Chapter 1 Introduction

1.1 Overview....9
1.2 System Features .... 11

Front View....11

Front View: SYS-221BT-HNR 12

Front View: SYS-221BT-HNTR....13

Front View: SYS-221BT-HNC8R....14

Front View: SYS-221BT-HNC9R....15

Control Panel....16

Rear View....17

1.3 System Architecture ....18

Main Components....18

System Block Diagram....19

1.4 Motherboard Layout....23

Quick Reference Table....24

Chapter 2 Server Installation

2.1 Overview....26
2.2 Unpacking the System 26
2.3 Preparing for Setup....26

Choosing a Setup Location....26

Rack Precautions....27

Server Precautions....27

Rack Mounting Considerations 27

Ambient Operating Temperature....27
Airflow 28
Mechanical Loading....28
Circuit Overloading....28
Reliable Ground....28

2.4 Rack Mounting Instructions....30

Overview of the Rack Rails....30

Adjusting the Rail Length ....30

Installing the Rails on a Rack....31

Chassis Installation 32

Chapter 3 Maintenance and Component Installation

3.1 Removing Power ....33

3.2 Accessing the System....34

Removing a Computing Node Drawer....34

Removing the Chassis Cover 35

3.3 Processor and Heatsink Installation....36

Installation Overview....37

Removal Overview....37

Create the Processor Carrier Assembly 38

Creating the PHM 42

Preparing the CPU Socket for Installation....44

Installing the PHM into the CPU Socket....45

Removing the PHM from the CPU Socket 46

Removing the Processor Carrier Assembly from the PHM 47

Removing the Processor from the Processor Carrier Assembly 48

3.4 Motherboard Installation....49

Tools Needed 49

Location of Mounting Holes 49

Installing the Motherboard....50

3.5 Memory....51

Memory Support....51

Memory Installation Sequence....51

General Memory Population Requirements....51

DDR5 Memory Population Guidelines ....52

DIMM Installation ....54

DIMM Removal 54

3.6 Motherboard Battery....55

3.7 Storage Drives....56

Drive Carriers....56

Drive Configuration ....57

Installing Drives....58

Hot-Swap for NVMe Drives....60

Checking the Temperature of an NVMe Drive 61

3.8 System Cooling....62

Fans 62

Installing the Air Shroud....64

3.9 Power Supply 65

3.10 AIOM Card 66

3.11 Expansion Cards....67

3.12 Backplane....72

3.13 Storage Adapters ....72

3.14 Cable Routing Diagrams....73

All Flash NVMe Gen5 73

NVMe and SATA 74

NVMe and SAS via onboard 3808....75

NVMe and SAS via 3908 AOC 76

Chapter 4 Motherboard Connections

4.1 Power Connections ....77

4.2 Headers and Connectors....78

4.3 Input/Output Ports 81

4.4 Jumpers....83

How Jumpers Work....83

4.5 LED Indicators....85

Chapter 5 Software

5.1 Microsoft Windows OS Installation....86

5.2 Driver Installation....88

5.3 SuperDoctor ^® 5....89

5.4 BMC....90

BMC ADMIN User Password 90

Chapter 6 Optional Components

6.1 Optional Parts List....91

6.2 TPM Security Module....91

6.3 Intel Virtual RAID on CPU (VROC)....92

Requirements and Restrictions....92

Supported SSDs and Operating Systems 92

Additional Information 93

Hardware Key 93

Configuring Intel VMD....94

Configuring VMD Manually....94

Creating NVMe RAID Configurations....98

Status Indications....99

Hot-Swap Drives 99

Hot-unplug 99

Hot-plug 99

Related Information Links 99

Chapter 7 Troubleshooting and Support

7.1 Information Resources ....100

Website 100

Direct Links for the SYS-221BT-H Series System 100

Direct Links for General Support and Information 101

7.2 Baseboard Management Controller (BMC)....101

7.3 Troubleshooting Procedures .....102

General Technique....102

No Power 102

No Video 103

System Boot Failure 103

Memory Errors 103

Losing the System Setup Configuration 103

When the System Becomes Unstable 103

7.4 Crash Dump Using BMC....105

7.5 UEFI BIOS Recovery 106

Overview 106

Recovering the UEFI BIOS Image....106

Recovering the Main BIOS Block with a USB Device....106

7.6 CMOS Clear 111

7.7 Where to Get Replacement Components....112

7.8 Reporting an Issue 112

Technical Support Procedures 112

Returning Merchandise for Service....112

Vendor Support Filing System 113

7.9 Feedback....113

Appendix A Standardized Warning Statements for AC Systems Appendix B System Specifications

BSMI/RoHS 137

Contacting Supermicro

Headquarters

Address: Super Micro Computer, Inc.

980 Rock Ave.

San Jose, CA 95131 U.S.A.

Tel: [phone]

Fax: [phone]

Email: [e-mail] (General Information)

[e-mail] (Sales Inquiries)

[e-mail] (Gov. Sales Inquiries)

[e-mail] (Technical Support)

[e-mail] (RMA Support)

[e-mail] (Webmaster)

Website: www.supermicro.com

Europe

Address: Super Micro Computer B.V.

's-Hertogenbosch, The Netherlands

Tel: [phone]

Fax: [phone]

Email: [e-mail] (Sales Inquiries)

[e-mail] (Technical Support)

[e-mail] (RMA Support)

Website: www.supermicro.nl

Asia-Pacific

Address: Super Micro Computer, Inc.

3F, No. 150, Jian 1st Rd.

Zhonghe Dist., New Taipei City 235

Taiwan (R.O.C)

Tel: [phone]

Fax: [phone]

Email: [e-mail] (Sales Inquiries)

[e-mail] (Technical Support)

[e-mail] (RMA Support)

Website: www.supermicro.com.tw

Chapter 1

Introduction

1.1 Overview

This chapter provides a brief outline of the functions and features of the SuperServer SYS-221BT-H Series. It provides four nodes for efficient power sharing, OCP 3.0 interfaces, and an optional M.2 NVMe physical RAID controller. The SuperServer SYS-221BT-H Series is based on the X13DET-B motherboard and the CSE-217BQ2-R3K04P chassis.

System Overview
Motherboard	X13DET-B
Chassis	CSE-217BQ2-R3K04P
Processor Support	Supports dual 4th Gen Intel Xeon Scalable Processors (in Socket E LGA4677) with four UPIs (max 16GT/s) and a TDP (thermal design power) of up to 350 W per node.Supports SP XCC, SP MCC, and HBM SKU on the X13DET-B.
Chipset	Intel C741 PCH
Memory	Supports up to 4 TB RDIMM DDR5 (288-pin) ECC memory with speeds up to 4800 MT/s in 16 DIMM slots
Drive Support	Front hot-swappable drives include:24 NVMe drives for SYS-221BT-HNR (all drives are PCIe 5.0)24 NVMe/SATA drives for SYS-221BT-HNTR (the first two drives are PCIe 5.0 in each node)24 NVMe/SAS drives for SYS-221BT-HNC8R (the first two drives are PCIe 5.0 in each node)24 NVMe/SAS drives for SYS-221BT-HNC9R (the first two drives are PCIe 5.0 in each node)Internal M.2 drives per node includes one of the following:Carrier with two M.2 NVMe Gen 4 SSDs in the 22110 form factorCarrier with two M.2 NVMe Gen 3 SSDs in the 2280 form factorInternal Connector:VROC key header
Expansion Slots	Two PCIe 5.0 x16 slots for each node
Networking	One AIOM or any compliant OCP 3.0 SFF Network Interface Card per nodeOne dedicated LAN port for BMC per node
I/O Ports	Two USB 3.1 ports per nodeOne VGA port per nodeOne internal COM port per node
System Cooling	Four 8-cm mid-chassis fansTwo CPU heatsinks per nodeOne air shroud per node
Power	Redundant 3000 W modules, 80Plus level Titanium
Form Factor	2U rackmount; (WxHxD) 17.6 x 3.5 x 28.8 in (447 x 88 x 730 mm)

Notes: A Quick Reference Guide can be found on the product page of the Supermicro website. The following safety models associated with the SYS-221BT-H Series have been certified as compliant with CSA or UL models: 217B-22, 217B-30, 217B-R22X13, 217B-R30X13.

System Models
Models Storage Capabilities (per node)
SYS-221BT-HNR Six N	VMe drives via backplane and NVMe cables
SYS-221BT-HNTR Six	hybrid NVMe/SATA drives via backplane and NVMe and SATA cables
SYS-221BT-HNC8R Six	hybrid NVMe/SAS drives via backplane and NVMe cables and 3808 SAS card
SYS-221BT-HNC9R	Six hybrid NVMe/SAS drives via backplane and NVMe cables, riser card, and 3908 add-on card

Note: Of the six hybrid drives, two support PCIe 5.0 NVMe and four support PCIe 4.0 NVMe. The SYS-221BT-HNR supports PCIe 5.0 NVMe on all slots.

1.2 System Features

The CSE-217BQ2-R3K04P is a 2U chassis that supports 24 front hot-swappable drives and four rear hot-pluggable nodes.

Front View

The chassis front offers access to the storage drives, a control panel for each node, and a pull-out service tag.

Control Panels for Node A (bottom) and Node B (top) Control Panels for Node C (bottom) and Node D (top) Service Tag Thumbscrew C3 C5C4 D0 D2D1 D3 D5D4 Thumbscrew

Figure 1-1. Front View

System Features: Front
Feature Description
Control Panels	Four control panels with labels for each node. The four control panels are located as follows: node A bottom left, node B top left, node C bottom right, and node D top right.
Service Tag Pull-out service tag with BMC password label
Drive Bays 24 2.5" hot-swappable drive bays
Thumbscrews Two thumbscrews to secure the server onto the rack

Front View: SYS-221BT-HNR

Power Button/LED (Node B) Node A Node A A0 A1 A2 A3 A4 A5 B0 B1 B2 B3 B4 B5 C0 C1 C2 C3 C4 C5 D0 D1 D2 D3 D4 D5 Node B Node C Node D (Node C) -(Node A) -(Node A) UID Button/LED Information LEDs -(Node B) -(Node D)

Figure 1-2. Front View

Drive Bays
Drive Bays (Node A - Node D) Description
	Two 2.5" Hot-Swap NVMe Gen 5 (CPU1) Drive Bays
	Four 2.5" Hot-Swap NVMe Gen 5 (CPU2) Drive Bays

Front View: SYS-221BT-HNTR

Power Button/LED (Node B) Node A Node A A0 A1 A2 A3 A4 A5 B0 B1 B2 B3 B4 B5 C0 C1 C2 C3 C4 C5 D0 D1 D2 D3 D4 D5 Node B Node C Node D (Node C) -(Node A) -(Node A) UID Button/LED Information LEDs

Figure 1-3. Front View

Drive Bays
Drive Bays (Node A - Node D) Description
0 1	Two 2.5" Hot-Swap NVMe Gen 5 (CPU1) Drive Bays
2 5	Four 2.5" Hot-Swap NVMe Gen 5 (CPU2) Drive Bays

Front View: SYS-221BT-HNC8R

Power Button/LED (Node B) Node A Node A A0 A1 A2 A3 A4 A5 B0 B1 B2 B3 B4 B5 C0 C1 C2 C3 C4 C5 D0 D1 D2 D3 D4 D5 Node B Node C Node D (Node C) -(Node A) -(Node A) UID Button/LED Information LEDs

Figure 1-4. Front View

Drive Bays
Drive Bays (Node A - Node D) Description
0 1	Two 2.5" Hot-Swap NVMe Gen 5 (CPU1) Drive Bays
2 5	Four 2.5" Hot-Swap NVMe Gen 5 (CPU2) Drive Bays

Front View: SYS-221BT-HNC9R

Power Button/LED (Node B) Node A Node B Node C Node D (Node C) Node A Node B Node C Node D (Node D) Node A Node B Node C Node D Node A Node B Node C Node D Node A Node B Node C Node D Node A Node B Node C Node D Node A Node B Node C Node D Node A Node B Node C Node D Node A Node B Node C Node D Node A Node B Node C Node D

Figure 1-5. Front View

Drive Bays
Drive Bays (Node A - Node D) Description
0 1	Two 2.5" Hot-Swap NVMe Gen 5 (CPU1)/SAS Drive Bays
2 5	Four 2.5" Hot-Swap NVMe Gen 4 (CPU2)/SAS Drive Bays

Control Panel

NIC LED Information LED Node Label Power Button UID LED Nucleus B SUPERXIMIC

Figure 1-6. Control Panel (One Per Node)

Control Panel Features (One Per Node)
Feature Description
Power Button	The main power switch applies or removes primary power from the power supply to the server but maintains standby power.
NIC LED Indicates	network activity on the LAN when flashing.
Information LED Universal information LED (see table below for details).
UID LED	The unit identification (UID) button turns on or off the blue light function of the Information LED and a blue LED on the rear of the chassis. These are used to locate the server in large racks and server banks.
Node Label	Label with the name of the node that is connected to the control panel. Labels for nodes A and C are above their control panel. Labels for nodes B and D are below their control panel.

Information LED
Color, Status Description
Red, solid An overheat condition has occurred.
Red, blinking at 1 Hz Fan failure, check for an inoperative fan.
Red, blinking at 0.25 Hz	Power failure, check for a non-operational power supply.
Red, solid, with Power LED blinking green Fault detected
Blue and red, blinking at 10 Hz Recovery mode
Blue, solid	UID has been activated locally to locate the server in a rack environment.
Blue, blinking at 1 Hz	UID has been activated using the BMC to locate the server in a rack environment.
Blue, blinking at 2 Hz BMC is resetting
Blue, blinking at 4 Hz BMC is setting factory defaults
Blue, blinking at 10 Hz with Power LED blinking green	BMC/BIOS firmware is updating

Rear View

Node D Node Handle Power Supply Node B 2 1 A1 2 1 Node C Dual Function Switch BMC LAN USB Ports VGA Port 2 1 A1 2 1 Node A

Figure 1-7. System: Rear View

System Features: Rear
Feature Description
Nodes A, B, C, D Independent computing nodes
Power Supplies Two 3000 W redundant power supplies
VGA Port Video port
USB Ports Two USB 3.1 ports
LAN Port Dedicated BMC LAN port
Dual Function Switch	A switch that can function as either a UID LED switch or a BMC reset switch. See Chapter 4 for a description of the dual function switch.
A1	x16 AIOM slot (CPU1) in each node

Expansion Slot Locations

Item Description (all PCIe 5.0)

1 x16 low profile slot (CPU1) in each node

2 x16 low profile slot (CPU2) in each node

CPU1 CPU2

1.3 System Architecture

This section covers the printed circuit board (PCB) locations and the system block diagrams.

Main Components

Storage Backplane M.2 Card Carrier Storage Adapter SAS Card for SYS-221BT-HNC8R Motherboard AIOM Riser Card for SYS-221BT-HNC9R AOC SAS3908

Figure 1-8. SYS-221BT-HNTR Board Locations

System Block Diagram

The block diagram below shows the connections and relationships between the subsystems and major components of the overall system.

graph TD
    subgraph SYS-221BT-HNR Block Diagram
        A["DDR5 4800"] -->|DIMM1| B["DDR5 4800"]
        B -->|DIMM1| C["DDR5 4800"]
        C -->|DIMM1| D["DDR5 4800"]
        D -->|DIMM1| E["DDR5 4800"]
        E -->|DIMM1| F["DDR5 4800"]
        F -->|DIMM1| G["DDR5 4800"]
        G -->|DIMM1| H["DDR5 4800"]
        H -->|DIMM1| I["DDR5 4800"]
        I -->|DIMM1| J["DDR5 4800"]
        J -->|DIMM1| K["DDR5 4800"]
        K -->|DIMM1| L["DDR5 4800"]
        L -->|DIMM1| M["DDR5 4800"]
        M -->|DIMM1| N["DDR5 4800"]
        N -->|DIMM1| O["DDR5 4800"]
        O -->|DIMM1| P["DDR5 4800"]
        P -->|DIMM1| Q["DDR5 4800"]
        Q -->|DIMM1| R["DDR5 4800"]
        R -->|DIMM1| S["DDR5 4800"]
        S -->|DIMM1| T["DDR5 4800"]
        T -->|DIMM1| U["DDR5 4800"]
        U -->|DIMM1| V["DDR5 4800"]
        V -->|DIMM1| W["DDR5 4800"]
        W -->|DIMM1| X["DDR5 4800"]
        X -->|DIMM1| Y["DDR5 4800"]
        Y -->|DIMM1| Z["DDR5 4800"]
        Z -->|DIMM1| AA["DDR5 4800"]
        AA -->|DIMM1| AB["DDR5 4800"]
        AB -->|DIMM1| AC["DDR5 4800"]
        AC -->|DIMM1| AD["DDR5 4800"]
        AD -->|DIMM1| AE["DDR5 4800"]
        AE -->|DIMM1| AF["DDR5 4800"]
        AF -->|DIMM1| AG["DDR5 4800"]
        AG -->|DIMM1| AH["DDR5 4800"]
        AH -->|DIMM1| AI["DDR5 4800"]
        AI -->|DIMM1| AJ["DDR5 4800"]
        AJ -->|DIMM1| AK["DDR5 4800"]
        AK -->|DIMM1| AL["DDR5 4800"]
        AL -->|DIMM1| AM["DDR5 4800"]
        AM -->|DIMM1| AN["DDR5 4800"]
        AN -->|DIMM1| AO["DDR5 4800"]
        AO -->|DIMM1| AP["DDR5 4800"]
        AP -->|DIMM1| AQ["DDR5 4800"]
        AQ -->|DIMM1| AR["DDR5 4800"]
        AR -->|DIMM1| AS["DDR5 4800"]
        AS -->|DIMM1| AT["DDR5 4800"]
        AT -->|DIMM1| AU["DDR5 4800"]
        AU -->|DIMM1| AV["DDR5 4800"]
        AV -->|DIMM1| AW["DDR5 4800"]
        AW -->|DIMM1| AX["DDR5 4800"]
        AX -->|DIMM1| AY["DDR5 4800"]
        AY -->|DIMM1| AZ["DDR5 4800"]
        AZ -->|DIMM1| BA["DDR5 4800"]
        BA -->|DIMM1| BB["DDR5 4800"]
        BB -->|DIMM1| BC["DDR5 4800"]
        BC -->|DIMM1| BD["DDR5 4800"]
        BD -->|DIMM1| BE["DDR5 4800"]
        BE -->|DIMM1| BF["DDR5 4800"]
        BF -->|DIMM1| BG["DDR5 4800"]
        BG -->|DIMM1| BH["DDR5 4800"]
        BH -->|DIMM1| BI["DDR5 4800"]
        BI -->|DIMM1| BJ["DDR5 4800"]
        BJ -->|DIMM1| BK["DDR5 4800"]
        BK -->|DIMM1| BL["DDR5 4800"]
        BL -->|DIMM1| BM["DDR5 4800"]
        BM -->|DIMM1| BN["DDR5 4800"]
        BN -->|DIMM1| BO["DDR5 4800"]
        BO -->|DIMM1| BP["DDR5 4800"]
        BP -->|DIMM1| BQ["DDR5 4800"]
        BQ -->|DIMM1| BR["DDR5 4800"]
        BR -->|DIMM1| BS["DDR5 4800"]
        BS -->|DIMM1| BT["DDR5 4800"]
        BT -->|DIMM1| BU["DDR5 4800"]
        BU -->|DIMM1| BV["DDR5 4800"]
        BV -->|DIMM1| BW["DDR5 4800"]
        BW -->|DIMM1|
        AX --> CY["XSB3"] --> CY
        CY --> CYx["XSB4"] --> CYx["XSB3"] --> CYxX["XSB3"] --> CYxXx["XSB3"] --> CYxXxX["XSB3"] --> CYxXxX["XSB3"] --> CYxXxX["XSB3"] --> CYxXxX["XSB3"] --> CYxXxX["XSB3"] --> CYxXxX["XSB3"] --> CYxXxX["XSB3"] --> CYxXxX["XSB3"] --> CYxXxX["XSB3"] --> CYxXxX["XSB3"] --> CYxXxS["XSB3"] & CYxS["XSB3"] & CYxS["XSB3"] & CYxS["XSB3"] & CYxS["XSB3"] & CYxS["XSB3"] & CYxS["XSB3"] & CYxS["XSB3"] & CYxS["XSB3"] & CYxS["XSB3"] & CYxS["XSB3"] & CYxS["XSB3"] & CYxS["XSB3"] & CYxS["X SB3"] & CYxS["X SB3"] & CYxS["X SB3"] & CYxS["X SB3"] & CYxS["X SB3"] & CYxS["X SB3"] & CYxS["X SB3"] & CYxS["X SB3"] & CYxS["X SB3"] & CYxS["X SB3"] & CYxS["X SB3"] & CYxS["X SB3"] & CYxS["X SB3"] & CXE[x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x 
    end
    subgraph PCS_x6
    PCle_x6
    PCle_x7
    PCle_x7
    PCle_x7
    PCle_x7
    PCle_x7
    PCle_x7
    PCle_x7
    PCle_x7
    PCle_x7
    PCle_x7
    PCle_x7
    PCle_x7
    PCle_x7
    PCle_x7
    PCle_x7
    PCle_x7
    PCle_x7
    PCLe_x6
    PCLe_x6
    PCLe_x6
    PCLe_x6
    PCLe_x6
    PCLe_x6
    PCLe_x6
    PCLe_x6
    PCLe_x6
    PCLe_x6
    PCLe_x6
    PCLe_x6
    PCLe_x6
    PCLe_x6
    PCLe_x6
    PCLe_x6
    PCLe_x6
    PCLE_x6
    PCLE_x6
    PCLE_x6
    PCLE_x6
    PCLE_x6
    PCLE_x6
    PCLE_x6
    PCLE_x6
    PCLE_x6
    PCLE_x6
    PCLE_x6
    PCLE_x6
    PCLE_x6
    PCLE_x6
    PCLE_x6
    PCLE_x6
    PCLE_x6

Figure 1-9. SYS-221BT-HNR System Block Diagram

Note: The figure is for illustrative purposes only. See the memory section and expansion card section for more details about possible configurations.

```mermaid
graph TD
    subgraph SYS-221BT-HNTR Block
        direction TB
        A["DDR5 4800"] --> B["DDR5 4800"]
        B --> C["DDR5 4800"]
        C --> D["DDR5 4800"]
        D --> E["DDR5 4800"]
        E --> F["DDR5 4800"]
        F --> G["DDR5 4800"]
        G --> H["DDR5 4800"]
        H --> I["DDR5 4800"]
        I --> J["DDR5 4800"]
        J --> K["DDR5 4800"]
        K --> L["DDR5 4800"]
        L --> M["DDR5 4800"]
        M --> N["DDR5 4800"]
        N --> O["DDR5 4800"]
        O --> P["DDR5 4800"]
        P --> Q["DDR5 4800"]
        Q --> R["DDR5 4800"]
        R --> S["DDR5 4800"]
        S --> T["DDR5 4800"]
        T --> U["DDR5 4800"]
        U --> V["DDR5 4800"]
        V --> W["DDR5 4800"]
        W --> X["DDR5 4800"]
        X --> Y["DDR5 4800"]
        Y --> Z["DDR5 4800"]
        Z --> AA["DDR5 4800"]
        AA --> AB["DDR5 4800"]
        AB --> AC["DDR5 4800"]
        AC --> AD["DDR5 4800"]
        AD --> AE["DDR5 4800"]
        AE --> AF["DDR5 4800"]
        AF --> AG["DDR5 4800"]
        AG --> AH["DDR5 4800"]
        AH --> AI["DDR5 4800"]
        AI --> AJ["DDR5 4800"]
        AJ --> AK["DDR5 4800"]
        AK --> AL["DDR5 4800"]
        AL --> AM["DDR5 4800"]
        AM --> AN["DDR5 4800"]
        AN --> AO["DDR5 4800"]
        AO --> AP["DDR5 4800"]
        AP --> AQ["DDR5 4800"]
        AQ --> AR["DDR5 4800"]
        AR --> AS["DDR5 4800"]
        AS --> AT["DDR5 4800"]
        AT --> AU["DDR5 4800"]
        AU --> AV["DDR5 4800"]
        AV --> AW["DDR5 4800"]
        AW --> AX["DDR5 4800"]
        AX --> AY["DDR5 4800"]
        AY --> AZ["DDR5 4800"]
        AZ --> BA["DDR5 4800"]
        BA --> BB["DDR5 4800"]
        BB --> BC["DDR5 4800"]
        BC --> BD["DDR5 4800"]
        BD --> BE["DDR5 4800"]
        BE --> BF["DDR5 4800"]
        BF --> BG["DDR5 4800"]
        BG --> BH["DDR5 4800"]
        BH --> BI["DDR5 4800"]
        BI --> BJ["DDR5 4800"]
        BJ --> BK["DDR5 4800"]
        BK --> BL["DDR5 4800"]
        BL --> BM["DDR5 4800"]
        BM --> BN["DDR5 4800"]
        BN --> BO["DDR5 4800"]
        BO --> BP["DDR5 4800"]
        BP --> BQ["DDR5 4800"]
        BQ --> BR["DDR5 4800"]
        BR --> BS["DDR5 4800"]
        BS --> BT["DDR5 4800"]
        BT --> BU["DDR5 4800"]
        BU --> BV["DDR5 4800"]
        BV --> BW["DDR5 4800"]
        BW --> BX["DDR5 4800"]
        BX --> BY["DDR5 4800"]
        BY --> BZ["DDR5 4800"]
        BZ --> CA["DDR5 4800"]
        CA --> CB["DDR5 4800"]
        CB --> CC["DDR5 4800"]
        CC --> CD["DDR5 4800"]
        CD --> CE["DDR5 4800"]
        CE --> CF["DDR5 4800"]
        CF --> CG["DDR5 4800"]
        CG --> CH["DDR5 4800"]
        CH --> CI["DDR5 4800"]
        CI --> CJ["DDR5 4800"]
        CJ --> CK["DDR5 4800"]
        CK --> CL["DDR5 4800"]
        CL --> CM["DDR5 4800"]
        CM --> CN["DDR5 4800"]
        CN --> CO["DDR5 4800"]
        CO --> CP["DDR5 4800"]
        CP --> CQ["DDR5 4800"]
        CQ --> CR["DDR5 4800"]
        CR --> CS["DDR5 4800"]
        CS --> CT["DDR5 4800"]
        CT --> CU["DDR5 4800"]
        CU --> CV["DDR5 4800"]
        CV --> CW["DDR5 4800"]
        CW --> CX["DDR5 4800"]
        CX --> CY["DDR5 4800"]
        CY --> CZ["DDR5 4800"]
        CZ --> DA["DDR5 4800"]
        DA --> DB["DDR5 4800"]
        DB --> DC["DDR5 48O"]

Figure 1-10. SYS-221BT-HNTR System Block Diagram

Note: The figure is for illustrative purposes only. See the memory section and expansion card section for more details about possible configurations.

graph TD
    subgraph SYS-221BT-HNC8R BlockDiagram
        direction TB
        A["CPU#1"] -->|P0 P1 P2 P3| B["DDR5"]
        C["DDR5"] -->|P0 P1 P2 P3| D["DDR5"]
        E["PCIe x8"] --> F["SCC-B8SB80-B1"]
        G["PCIe x16"] --> H["SCC-P6N6SGH-B1"]
        I["PCIe x8"] --> J["BPN-NVME5-217BHQ-S6"]
    end

    subgraph PLCs
        K["PCIe X8"] --> L["SCC-B8SB80-B1"]
        M["PCIe X16"] --> N["SCC-P6N6SGH-B1"]
    end

    subgraph Components
        O["AOM"] --> P["PCH 16W"]
        Q["USB 2.0"] --> P
        R["4x SATA3"] --> S["SATA1"]
        T["4x SATA3"] --> U["SATA2"]
        V["1.2x SATA3"] --> W["SXB5"]
        X["USB 2.0"] --> Y["USB Header"]
        Z["USB 3.1"] --> AA["USB Header"]
        AB["USB 3.0"] --> AC["USB Header"]
        AD["USB 3.1"] --> AE["USB Header"]
        AF["USB 3.0"] --> AG["USB Header"]
        AH["USB 3.1"] --> AI["USB Header"]
        AJ["USB 3.0"] --> AK["USB Header"]
        AL["USB 3.1"] --> AM["USB Header"]
        AN["USB 3.0"] --> AO["USB Header"]
        AP["USB 3.1"] --> AQ["USB Header"]
        AR["USB 3.0"] --> AS["USB Header"]
        AT["USB 3.1"] --> AU["USB Header"]
        AV["USB 3.0"] --> AW["USB Header"]
        AX["USB 3.1"] --> AY["USB Header"]
        AZ["USB 3.0"] --> BA["USB Header"]
        BB["USB 3.1"] --> BC["USB Header"]
        BD["USB 3.0"] --> BE["USB Header"]
        BF["USB 3.1"] --> BG["USB Header"]
        BH["USB 3.0"] --> BH
        BI["MUX"] --> BJ["BIOS"]
    end

    subgraph Components
        BK["NCSI Header"] --> BL["BMC AST2600"]
        BL --> BM["NCS master"]
        BM --> BN["LAN3 RTL821F"]
        BN --> BO["RGRMII"]
        BP["RJ46"] --> BP
        BP --> BP
        BP --> BP
        BP --> BP
        BP --> BP
        BP --> BP
        BP --> BP
        BP --> BP
        BP --> BP
        BP --> BP
        BP --> BP
        BP --> BP
        BP --> BP
        BP --> BP
        BP --> BP
        BP --> BP
        BP --> BP
        BP --> BP
        BP --> BP
        BP --> BP
        BP --> BP
        NP["MOMC"] --> NP
        NP --> NP
        NP --> NP
        NP --> NP
    end

    subgraph Components
        O
        BR["NCSI Header"] --> BS["BMC AST2600"]
        BS --> BT["NCS master"]
        BT --> BU["LAN3 RTL821F"]
        BU --> BV["RGRMII"]
        BV --> BW["RJR46"]
        BW --> BX["VGA CONX"]
        BX --> BY["COM1 Header"]
        BY --> BZ["Femo Sensor NCT7719W"]
        BZ --> CA["eVMC"]
    end

    subgraph Components
        SP["BMC Flash"] --> SP
        SP --> SP
        SP --> SP
        SP --> SP
        SP --> SP
    end

    subgraph Components
        AU["BMC TPW Onboard"] --> AU
    end

    subgraph Components
        AV["TIPM HEADER"] --> AV
    end

    subgraph Components
        AX["TIPM HEADER"] --> AX
    end

    style SYS-221BT-HNC8R BlockDiagram fill:#f9f9f9,stroke:#333,stroke-width:2px

Figure 1-11. SYS-221BT-HNC8R System Block Diagram

Note: The figure is for illustrative purposes only. See the memory section and expansion card section for more details about possible configurations.

SYS-221BT-HNC9R Block Diagram showing CPU, DDR, I/O, and peripheral components with signal paths and data flow between BPN-NVME5-217BHQ-S6.

Figure 1-12. SYS-221BT-HNC9R System Block Diagram

Note: The figure is for illustrative purposes only. See the memory section and expansion card section for more details about possible configurations.

1.4 Motherboard Layout

Below is a layout of the X13DET-B motherboard with jumper, connector and LED locations shown. See the table on the following page for descriptions. For detailed descriptions, pinout information and jumper settings, refer to Chapter 4 or the Motherboard Manual.

P2-D1MMD1 P2-D1MMC1 P2-D1MMB1 P2-D1MMA1 P2-D1MMH1 P2-D1MMF1 P2-D1MMG1 P2-D1MMH1 SXB1 JPWR1 JPWR2 JPWR3 JPWR4 P2 SXB1 PCIe 5.0 x16 P2 SXB3 PCIe 5.0 x8 P1 SXB4 PCIe 5.0 x8 P1-SXB3 PCIe 5.0 x8 P1-SXB4 PCIe 5.0 x8 P1-SXB5 PCIe 5.0 x8 M.2 SATA0~1 SXB5 P1-SXB5 PCIe 5.0 x8 P1-D1MMH1 P1-D1MMG1 P1-D1MMF1 P1-D1MME1 JBT1 SATA0~7 Slot2A JBR1 JPME2 JP1 Battery JBR1 JPM2 COM1 FAN3 JVRM1 SATA8~11 (SATA2) JCP LD1 P1 Slot1 PCIe 5.0 x16 Slot1 BMC JP6 JPME1 JPFR3 LE6 VGA USB0/1 UID PFR2 IPMI LAN UID_LED1/LEDBMC

Figure 1-13. Motherboard Layout

Quick Reference Table

Jumper Description Jumper Setting (Bold: Default)

JBT1 CMOS Clear Open (Normal)

JPME1 ME Recovery Pins 1/2: Normal, Pins 2/3: ME Recovery

JPME2 ME (Manufacture Mode) Select Pins 1/2: ME Mode

LED Description Status

LE1 CPLD Heartbeat LED On: Complex Programmable Logical Device (CPLD) Normal (Active)
LE6 Power Status LED		Green: Power OnAmber: Standby PowerRed: Power Failure
UID_LED1/LEDBMC	Rear Unit Identifier (UID)LED/BMC Heartbeat LED	Solid Blue (working as a UID LED): Unit IdentifiedBlinking Green (working as a BMC LED): BMC Normal (Active)Solid Green: During BMC Reset or during a Cold Reboot

Connector Description

Battery (BT1) Onboard battery
BMC_LAN (IPMI_LAN1)	Dedicated BMC LAN port on the I/O backplane
COM1 (JCOM1)	Serial/COM port header
JAIOM	Supermicro Advanced input/output Module (AIOM) PCIe 5.0 x16 slot supported by CPU1
JNCSI1	NC-SI (Network Controller Sideband Interface) connector (See Note 1 on the next page.)
JTPM1	Trusted Platform Module/Port 80 connector
JPWR1~JPWR4	6-pin power connectors (1~4)
NS1	P1 NS1 PCIe 5.0 x8 connector supported by CPU1
NS2	P2 NS2 PCIe 5.0 x8 connector supported by CPU2
NS3	P2 NS3 PCIe 5.0 x8 connector supported by CPU2
Slot1	P1 Slot1 PCIe 5.0 x16 supported by CPU1
Slot2 + Slot2A	P2 Slot2/Slot2A PCIe 5.0 x16 to support riser card(s)
SXB1	P2 SXB1 PCIe 5.0 x16 supported by CPU2
SXB3	P1 SXB3 PCIe 5.0 x8 supported by CPU1
SXB4	P1 SXB4 PCIe 5.0 x8 supported by CPU1
SXB5 (NVMe M.2)	P1 SXB5 PCIe 5.0 x8 for two NVMe M.2 devices in the M-Key 2280 or 22110 format (depending on the type of riser card used) supported by CPU1 (See Note 2 on the next page.)
SATA 0~7 (SATA1)	SATA connector with support of eight Intel® PCH SATA 3.0 connections (RAID 0, RAID 1, RAID 5, and RAID 10 supported)
SATA 8~11 (SATA2)	SATA connector with four SATA connections (SATA 8~11)
UID Button (JUIDB1)	Unit Identifier/BMC Reset switch/button on the rear I/O panel
USB0/1 (3.1) (JUSB1)	USB 3.1 connector on the rear I/O panel for two USB parts (Port0/Port1)
VGA (JVGA1)	Rear VGA connector

VROC RAID Key (JRK1) Intel VROC key header for NVMe RAID support (See Note 3 on the next page.)

Note 1: For details on how to configure Network Interface Card (NIC) settings, please refer to the Network Interface Card Configuration User's Guide posted on our website under the link: http://www.supermicro.com/support/manuals/.

Note 2: The SXB5 slot supports the SCC-P2HM2G4-B1 and the SCC-A2NM2241G3-B1. The SCC-P2HM2G4-B1 supports two NVMe M.2 devices (up to Gen 4) in the 22110 form factor, and the SCC-A2NM2241G3-B1 supports two NVMe M.2 devices (up to Gen 3) in the 2280 form factor.

Note 3: For detailed instructions on how to configure VROC RAID settings, please refer to the VROC RAID Configuration User's Guide posted on the web page under the link: http://www.supermicro.com/support/manuals/

Chapter 2

Server Installation

2.1 Overview

This chapter provides advice and instructions for mounting your system in a server rack. If your system is not already fully integrated with processors, system memory etc., refer to Chapter 3 for details on installing those specific components.

Caution: Electrostatic Discharge (ESD) can damage electronic components. To prevent such damage to PCBs (printed circuit boards), it is important to use a grounded wrist strap, handle all PCBs by their edges and keep them in anti-static bags when not in use.

2.2 Unpacking the System

Inspect the box in which the SuperServer SYS-221BT-H Series was shipped, and note if it was damaged in any way. If any equipment appears damaged, file a damage claim with the carrier who delivered it.

Decide on a suitable location for the rack unit that will hold the server. It should be situated in a clean, dust-free area that is well ventilated. Avoid areas where heat, electrical noise and electromagnetic fields are generated. It will also require a grounded AC power outlet nearby. Be sure to read the precautions and considerations noted in Appendix A.

2.3 Preparing for Setup

The box in which the system was shipped should include the rackmount hardware needed to install it into the rack. Please read this section in its entirety before you begin the installation.

Choosing a Setup Location

• The system should be situated in a clean, dust-free area that is well ventilated. Avoid areas where heat, electrical noise and electromagnetic fields are generated.
• Leave enough clearance in front of the rack so that you can open the front door completely (\~25 inches) and approximately 30 inches of clearance in the back of the rack to allow sufficient space for airflow and access when servicing.
• This product should be installed only in a Restricted Access Location (dedicated equipment rooms, service closets, etc.).

- This product is not suitable for use with visual display workplace devices according to §2 of the German Ordinance for Work with Visual Display Units.

Rack Precautions

• Ensure that the leveling jacks on the bottom of the rack are extended to the floor so that the full weight of the rack rests on them.
• In single rack installations, stabilizers should be attached to the rack. In multiple rack installations, the racks should be coupled together.
• Always make sure the rack is stable before extending a server or other component from the rack.
• You should extend only one server or component at a time - extending two or more simultaneously may cause the rack to become unstable.

Server Precautions

• Review the electrical and general safety precautions in Appendix A.
• Determine the placement of each component in the rack before you install the rails.
• Install the heaviest server components at the bottom of the rack first and then work your way up.
• Use a regulating uninterruptible power supply (UPS) to protect the server from power surges and voltage spikes and to keep your system operating in case of a power failure.
• Allow any drives and power supply modules to cool before touching them.
• When not servicing, always keep the front door of the rack and all covers/panels on the servers closed to maintain proper cooling.
• To maintain proper cooling, always keep all chassis panels closed and all SATA carriers installed when not being serviced.

Rack Mounting Considerations

Ambient Operating Temperature

If installed in a closed or multi-unit rack assembly, the ambient operating temperature of the rack environment may be greater than the room's ambient temperature. Therefore, consideration should be given to installing the equipment in an environment compatible with the manufacturer's maximum rated ambient temperature (TMRA).

Airflow

Equipment should be mounted into a rack so that the amount of airflow required for safe operation is not compromised.

Mechanical Loading

Equipment should be mounted into a rack so that a hazardous condition does not arise due to uneven mechanical loading.

Circuit Overloading

Consideration should be given to the connection of the equipment to the power supply circuitry and the effect that any possible overloading of circuits might have on overcurrent protection and power supply wiring. Appropriate consideration of equipment nameplate ratings should be used when addressing this concern.

Reliable Ground

A reliable ground must be maintained at all times. To ensure this, the rack itself should be grounded. Particular attention should be given to power supply connections other than the direct connections to the branch circuit (i.e. the use of power strips, etc.).

To prevent bodily injury when mounting or servicing this unit in a rack, you must take special precautions to ensure that the system remains stable. The following guidelines are provided to ensure your safety:

• This unit should be mounted at the bottom of the rack if it is the only unit in the rack.
• When mounting this unit in a partially filled rack, load the rack from the bottom to the top with the heaviest component at the bottom of the rack.
• If the rack is provided with stabilizing devices, install the stabilizers before mounting or servicing the unit in the rack.
• Slide rail mounted equipment is not to be used as a shelf or a work space.

To prevent bodily injury when mounting or servicing this unit in a rack, you must take special precautions to ensure that the system remains stable. The following guidelines are provided to ensure your safety:

• This unit should be mounted at the bottom of the rack if it is the only unit in the rack.
• When mounting this unit in a partially filled rack, load the rack from the bottom to the top with the heaviest component at the bottom of the rack.
• If the rack is provided with stabilizing devices, install the stabilizers before mounting or servicing the unit in the rack.
• Slide rail mounted equipment is not to be used as a shelf or a work space.

Warning: Do not pick up the server with the front handles. They are designed to pull the system from a rack only.

Slide rail mounted equipment is not to be used as a shelf or a work space.

2.4 Rack Mounting Instructions

This section provides information on installing the chassis into a rack unit with the rails provided. There are a variety of rack units on the market, which may mean that the assembly procedure will differ slightly from the instructions provided. You should also refer to the installation instructions that came with the rack unit you are using. Note: This rail will fit a rack between 28" and 33.5" deep.

Overview of the Rack Rails

The package includes two rail assemblies. Each is specifically designed for the left or right side of the chassis, and so marked. Each rail consists of two sections: a front section which secures to the front post of the rack and a rear section which adjusts in length and secures to the rear post of the rack.

Rear Section

Figure 2-1. Rackmount Rail (Right rail assembly shown)

Adjusting the Rail Length

Each rail assembly has a locking screw to adjust the length of the rail to fit the depth of your rack.

Installing the Rails on a Rack

1. Loosen the adjusting screw to allow the rear section to slide in the front section.
2. Push the small hooks on the front section of the rail into the holes on the front post of the rack and then down, until the spring-loaded pegs snap into the rack holes. Secure the rail to the rack with screws.
3. Pull out the rear section of the outer rail, adjusting the length until it fits within the posts of the rack and align the small hooks with the appropriate holes on the rear post of the rack. Be sure the rail is level, then mount the rear section onto the rack. Secure the rail with screws.
4. Tighten the adjusting screw.

Technical line drawing of a server rack with labeled components (no text or symbols)

Figure 2-2. Attaching the Rail Front to the Rack (Left rail shown)

Note: Figures are for illustrative purposes only. Always install servers into racks from the bottom up.

Chassis Installation

Slide the chassis into the rack so that the bottom of the chassis slides onto the bottom lip of the rails.

Technical line drawing of a server rack with a device and mounting bracket, showing no text or symbols

Figure 2-3. Sliding the Chassis into the Rack

Stability hazard. The rack stabilizing mechanism must be in place, or the rack must be bolted to the floor before you slide the unit out for servicing. Failure to stabilize the rack can cause the rack to tip over.

Chapter 3

Maintenance and Component Installation

This chapter provides instructions on installing and replacing main system components. To prevent compatibility issues, only use components that match the specifications and/or part numbers given.

Installation or replacement of most components require that power first be removed from the system. Please follow the procedures given in each section.

3.1 Removing Power

Before performing some setup or maintenance tasks, use the following procedure to ensure that power has been removed from the system.

Removing Power from a Node

- Use the operating system to power down the node.

Removing Power from the System

1. Use the operating system to power down all nodes.
2. Grasp the head of each power cord and gently pull it out of the back of the power supply.
3. Disconnect the cords from the power strip or wall outlet.

3.2 Accessing the System

Removing a Computing Node Drawer

Node Handle Node Release Handle

Figure 3-1. Removing a Node Tray

Removing a Node

1. Use the operating system to power down the node.
2. Remove any cables attached to the node.
3. Pull down the node release handle (the handle with the lock and downward arrow icon) and use both handles to slide the node out the chassis rear.

Removing the Chassis Cover

You can access some chassis components, such as fans, by removing the cover.

Remove two screws

Figure 3-2. Removing the Chassis Cover

Removing the Chassis Cover

The chassis top cover can be lifted off after removing two screws.

Caution: Except for short periods of time, do not operate the server without the cover in place. It provides proper airflow to prevent overheating.

3.3 Processor and Heatsink Installation

The processor (CPU) and processor carrier should be assembled together first to form the processor carrier assembly. This assembly will be then attached to the heatsink to form the processor heatsink module (PHM) before being installed into the CPU socket. Before installation, be sure to perform the following steps below:

• Please carefully follow the instructions given on the previous page to avoid ESD-related damages.
• Unplug the AC power cords from all power supplies after shutting down the system.
• Check that the plastic protective cover is on the CPU socket, and none of the socket pins are bent. If they are, contact your retailer.
• When handling the processor, avoid touching or placing direct pressure on the LGA lands (gold contacts). Improper installation or socket misalignment can cause serious damage to the processor or CPU socket, which may require manufacturer repairs.
• Thermal grease is pre-applied on a new heatsink. No additional thermal grease is needed.
  • Refer to the Supermicro website for updates on processor and memory support.
• All graphics in this manual are for illustrations only. Your components may look different.

Note 1: The 4th Gen Intel® Xeon® Scalable Processor comes with three CPU SKUs: SP XCC, SP MCC, and HBM. However, CPU SKU support is motherboard-specific and not all SKUs are enabled on every X13 motherboard. Please refer to the specifications of your motherboard for CPU SKU support. Also, be sure to use processors of the same SKU on a multi-processor motherboard.

Note 2: The 4th Gen Intel® Xeon® Scalable Processor comes with three CPU SKUs, and each SKU supports a distinctive carrier. The SP XCC CPU supports Carrier E1A; HBM CPU supports Carrier E1C, and SP MCC, Carrier E1B.

Note 3: This installation guide provides instructions on how to install 1U heatsink on the X13 motherboards. Please note that heatsink support depends on the chassis used in your system. For heatsink support on your motherboard or in your system, please refer to your system specifications.

Installation Overview

After preparing the system and following ESD precautions, there are four steps to installing the processor and heatsink onto the motherboard.

1. Attach the processor to a plastic carrier to create the processor carrier assembly.
2. Attach the processor carrier assembly to the heatsink to create the processor heatsink module (PHM).
3. Prepare the socket for PHM installation.
4. Install the PHM.

Removal Overview

After preparing the system and following ESD precautions, there are three steps to removing the processor and heatsink from the motherboard.

1. Remove the PHM from the motherboard.
2. Remove the processor carrier assembly from the heatsink.
3. Remove the processor from the carrier.

Create the Processor Carrier Assembly

Process Carrier Assembly

1. Hold the processor with the gold pins (LGA lands) facing down. Locate the gold triangle at the corner of the processor and the corresponding hollowed triangle on the processor carrier as shown below. These triangles indicate the location of pin 1.

Pin 1 SP XCC Pin 1 Carrier E1A Carrier

Pin 1 HBM Pin 1 Pin 1 E1C

Pin 1 SP MCC Carrier E1B

The Processor with its Matching Carrier

1. Turn the processor over (with the gold pins up). Locate the CPU keys on the processor and the four latches on the carrier as shown below.

SP XCC CPU Key CPU Key Latch Latch Latch

HBM CPU Key Latch Latch CPU Key Latch

SP MCC CPU Key CPU Key Latch Latch Latch Latch

1. Locate the lever on the CPU socket and press it down as shown below.

Lever Carrier E1A Lever Carrier E1B

1. Using Pin 1 as a guide, carefully align the CPU keys (A and B) on the processor against the CPU keys on the carrier (a and b) as shown in the drawing below.

2. Once they are properly aligned, carefully insert the CPU into the carrier, making sure that the CPU is properly secured by latches 1, 2, 3, and 4.

SP XCC A CPU Key B Latch 1 a CPU Key 2 3 Latch b CPU Key

SP MCC CPU Key A CPU Key B Latch 4 a 1 b 3 Latch Latch CPU Key

HBM CPU Key A B CPU Key Latch a b 3 4 Latch Latch CPU Key Latch

1. After the processor is placed inside the carrier, examine the four sides of the processor, making sure that the processor is properly seated on the carrier.

SP XCC (Top View) Component View) CPU Carrier Assembly

HBM (Top View) (Component View) CPU Carrier Assembly

SP MCC (Top View) (Component View) CPU Carrier Assembly

Creating the PHM

After creating the processor carrier assembly, please follow the instructions below to mount the processor carrier into the heatsink to form the PHM.

Note: If this is a new heatsink, the thermal grease has been pre-applied on the underside. Otherwise, apply the proper amount of thermal grease.

1. Turn the heatsink over with the thermal grease, which is on the reverse side of the heatsink, facing up. Pay attention to the two triangle cutouts (A, B) located at the diagonal corners of the heatsink as shown in the drawing below.
2. Hold the processor carrier component side facing up, and locate the triangle on the CPU and the triangle on the carrier. (Triangle indicates Pin 1.)
3. Using Pin 1 as a guide, turn the processor carrier assembly over with the gold contacts facing up. Locate Pin 1 (A) on the processor and Pin 1 (a) on the processor carrier assembly.
4. Align the corner marked a on the processor carrier assembly against the triangle cutout A on the heatsink, and align the corners marked b, c, and d on the processor assembly against the corners marked B, C, and D on the heatsinks.
5. Once they are properly aligned, place the corners marked a, b, c, and d on the processor carrier assembly into the corners of the heatsink marked A, B, C, and D making sure that all plastic clips are properly attached to the heatsink.

CPU Carrier Assembly

(CPU Component Side and Heatsink Bottom Side)

Preparing the CPU Socket for Installation

This motherboard comes with a plastic protective cover installed on the CPU socket. Remove it from the socket by following the instructions below:

1. Press the tabs inward.

active cover from the socket

1. Pull up the protective cover from the socket.

Technical line drawing of a mechanical assembly with two views: top shows a rectangular component, bottom shows a cross-sectional view with bolted components and a red arrow indicating a specific feature (no text or symbols present)

Installing the PHM into the CPU Socket

1. Locate four threaded fasteners (a, b, c, d) on the CPU socket.

2. Align PEEK nut A, which is next to the triangle (Pin 1) on the heatsink, against threaded fastener a on the CPU socket. Then align PEEK nuts B, C, and D on the heatsink against threaded fasteners b, c, and d on the CPU socket, making sure that all PEEK nuts on the heatsink are properly aligned with the correspondent threaded fasteners on the CPU socket.

A, B, C, D: Peek Nut on the Heatsink a, b, c, d: Threaded Fastener on the CPU socket

1. Once they are aligned, gently place the heatsink on top of the CPU socket, making sure that each PEEK nut is properly attached to its corresponding threaded fastener.

Latched State

1. Press all four rotating wires outwards and make sure that the heatsink is securely latched onto the CPU socket.

2. With a T30 screwdriver, tighten all PEEK nuts in the sequence of A, B, C, and D with even pressure. To avoid damaging the processor or socket, do not use excessive force when tightening the PEEK nuts. (For best durability, 8in-lbf torque is recommended.)

3. Examine all corners of the heatsink to ensure that the PHM is firmly attached to the CPU socket.

Diagram showing a device being processed from a base to a top, with red arrows indicating motion direction (no text or symbols present)

Removing the PHM from the CPU Socket

Before removing the PHM from the motherboard, be sure to shut down the system and unplug the power cables from the power supply. Then follow the steps below:

1. Use a T30 screwdriver to loosen the four PEEK nuts on the heatsink in the sequence of

A, B, C, and D.

A B C D

1. Once the PEEK nuts are loosened from the CPU socket, press the rotating wires inwards to unlatch the PHM from the socket as shown in the drawings below.

Unlatched State

1. Gently pull the PHM upwards to remove it from the CPU socket.

Technical illustration of a heat exchanger housing with red arrows indicating upward motion (no text or symbols present)

Removing the Processor Carrier Assembly from the PHM

To remove the processor carrier assembly from the PHM, please follow the steps below:

1. Detach the four plastic clips (marked a, b, c, d) on the processor carrier assembly from the four corners of the heatsink (marked A, B, C, D) as shown in the drawings below.

1U Heatsink (View of Component Side and Heatsink Bottom Side)

Pin 1 a b c D A B C d

SP XCC

Pin 1 A B C D a b c d

HBM

Pin 1 a b c d B C D A

SP MCC

1. When all plastic clips are detached from the heatsink, remove the processor carrier assembly from the heatsink.

1U Heatsink (View of Component Side and Heatsink Bottom Side)

Technical diagram of a computer processor with internal components and red directional arrows indicating assembly or movement (no text or symbols present)

SP XCC HBM SP MCC

Technical diagram of a mechanical component with red arrows indicating assembly or force direction (no text or symbols present)

Technical diagram of a computer processor with two views showing internal components and red arrows indicating directional movement (no text or symbols present)

Removing the Processor from the Processor Carrier Assembly

Once you have removed the processor carrier assembly from the PHM, you are ready to remove the processor from the processor carrier by following the steps below.

1. Unlock the lever from its locked position and push the lever upwards to disengage the processor from the processor carrier as shown in the drawing on the right below.

Technical diagram showing a mechanical component being processed into a device, with an inset highlighting the lever mechanism.

CPU Carrier Assembly (carrier E1A)

CPU Server Assembly (server 2.1A) Lever

CPU Carrier Assembly (carrier E1B)

1. Once the processor is loosened from the carrier, carefully remove the processor from the processor carrier.

Note: Please handle the processor with care to avoid damaging the processor and its pins.

Isometric diagram of a microchip with internal circuitry and an upward arrow indicating motion (no text or symbols)

SP XCC HBM

Isometric technical diagram of a layered electronic component with a red upward arrow indicating motion or force (no text or symbols present)

Isometric diagram of a computer motherboard with an arrow indicating upward motion (no text or symbols present)

SP MCC

3.4 Motherboard Installation

All motherboards have standard mounting holes to fit different types of chassis. Make sure that the locations of all the mounting holes for both the motherboard and the chassis match. Although a chassis may have both plastic and metal mounting fasteners, metal ones are highly recommended because they ground the motherboard to the chassis. Make sure that the metal standoffs click in or are screwed in tightly.

Tools Needed

Phillips Screwdriver (1)

Phillips Screws (9)

Standoffs (9) (Only if Needed)

Line drawing of a mechanical component with threaded end and central shaft (no text or symbols)

Location of Mounting Holes

Note 1: To avoid damaging the motherboard and its components, please do not use a force greater than 8 lbf-in on each mounting screw during motherboard installation.

Note 2: Some components are very close to the mounting holes. Please take precautionary measures to avoid damaging these components when installing the motherboard to the chassis.

Note 3: If installing an M.2 long riser, instead of the standard Phillips Screw, there will need to be a Ball Pin Screw installed at the location specified on the motherboard layout.

Installing the Motherboard

1. Install the I/O shield into the back of the chassis, if applicable.
2. Locate the mounting holes on the motherboard. See the previous page for the location.

Chassis Chassis

1. Locate the matching mounting holes on the chassis. Align the mounting holes on the motherboard against the mounting holes on the chassis.

36 Motherboard Chassis 36 Motherboard Chassis

1. Install standoffs in the chassis as needed.
2. Install the motherboard into the chassis carefully to avoid damaging other motherboard components.
3. Using the Phillips screwdriver, insert a pan head #6 screw into a mounting hole on the motherboard and its matching mounting hole on the chassis.
4. Repeat Step 5 to insert #6 screws into all mounting holes.
5. Make sure that the motherboard is securely placed in the chassis.

Note: Images displayed are for illustration only. Your chassis or components might look different from those shown in this manual.

3.5 Memory

Memory Support

The X13DET-B motherboard supports up to 4 TB RDIMM DDR5 (288-pin) ECC memory with speeds up to 4800 MT/s in 16 DIMM slots (Note below).

Note: Memory speed and capacity support depend on the processors used in the system.

For validated memory, use our Product Resources page. Check the Supermicro website for possible updates to memory support.

Memory Installation Sequence

Memory for this motherboard is populated using the "Fill First" method. The DIMM slots with blue release tabs are considered the first DIMM of their channel, and those with white release tabs are the second of the channel. When installing memory modules, be sure to populate the memory slots with the blue release tabs first and then populate those with the white release tabs.

General Memory Population Requirements

1. Be sure to use the memory modules of the same type and speed on the motherboard. Mixing of memory modules of different types and speeds is not allowed.
2. Using unbalanced memory topology such as populating two DIMMs in one channel while populating one DIMM in another channel will result in reduced memory performance.
3. Populating memory slots with a pair of DIMM modules of the same type and size will result in interleaved memory, which will improve memory performance.

DDR5 Memory Population Guidelines

The following memory population table was created based on guidelines provided by Intel to support Supermicro motherboards.

DDR5 Memory Population Table for X13DP Motherboards (with 16 DIMMs installed)
1 CPU: Memory Population Sequence
1 CPU & 1 DIMM	P1-DIMMA1P1-DIMME1P1-DIMMB1P1-DIMMF1
1 CPU & 2 DIMMs	P1-DIMMA1 / P1-DIMMG1P1-DIMMC1 / P1-DIMME1
1 CPU & 4 DIMMs P1-DIMMA1 / P1-DIMMC1 / P1-DIMME1 / P1-DIMMG1
1 CPU & 6 DIMM	P1-DIMMA1 / P1-DIMMC1 / P1-DIMMD1 / P1-DIMME1 / P1-DIMMF1 / P1-DIMMG1P1-DIMMA1 / P1-DIMMB1 / P1-DIMMC1 / P1-DIMME1 / P1-DIMMG1 / P1-DIMMH1P1-DIMMB1 / P1-DIMMC1 / P1-DIMMD1 / P1-DIMME1 / P1-DIMMF1 / P1-DIMMH1P1-DIMMA1 / P1-DIMMB1 / P1-DIMMD1 / P1-DIMMF1 / P1-DIMMG1 / P1-DIMMH1
1 CPU & 8 DIMMs	P1-DIMMA1 / P1-DIMMB1 / P1-DIMMC1 / P1-DIMMD1 / P1-DIMME1 / P1-DIMMF1 / P1-DIMMG1 / P1-DIMMH1
2 CPUs:(Recommended)	Memory Population Sequence
2 CPUs & 2 DIMMs	CPU1: P1-DIMMA1, CPU2: P2-DIMMA1CPU1: P1-DIMME1, CPU2: P2-DIMME1CPU1: P1-DIMMB1, CPU2: P2-DIMMB1CPU1: P1-DIMMF1, CPU2: P2-DIMMF1
2 CPUs & 4 DIMMs	CPU1: P1-DIMMA1 / P1-DIMMG1, CPU2: P2-DIMMA1 / P2-DIMMG1CPU1: P1-DIMMC1 / P1-DIMME1, CPU2: P2-DIMMC1 / P2-DIMME1
2 CPUs & 8 DIMMs	CPU1: P1-DIMMA1 / P1-DIMMC1 / P1-DIMME1 / P1-DIMMG1CPU2: P2-DIMMA1 / P2-DIMMC1 / P2-DIMME1 / P2-DIMMG1
2 CPUs & 12 DIMMs	CPU1: P1-DIMMA1/P1-DIMMC1/P1-DIMMD1/P1-DIMME1/P1-DIMMF1/P1-DIMMG1CPU2: P2-DIMMA1/P2-DIMMC1/P2-DIMMD1/P2-DIMME1/P2-DIMMF1/P2-DIMMG1CPU1: P1-DIMMA1/P1-DIMMB1/P1-DIMMC1/P1-DIMME1/P1-DIMMG1/P1-DIMMH1CPU2: P2-DIMMA1/P2-DIMMB1/P2-DIMMC1/P2-DIMME1/P2-DIMMG1/P2-DIMMH1CPU1: P1-DIMMB1/P1-DIMMC1/P1-DIMMD1/P1-DIMME1/P1-DIMMF1/P1-DIMMH1CPU2: P2-DIMMB1/P2-DIMMC1/P2-DIMMD1/P2-DIMME1/P2-DIMMF1/P2-DIMMH1CPU1: P1-DIMMA1/P1-DIMMB1/P1-DIMMD1/P1-DIMMF1/P1-DIMMG1/P1-DIMMH1CPU2: P2-DIMMA1/P2-DIMMB1/P2-DIMMD1/P2-DIMMF1/P2-DIMMG1/P2-DIMMH1
2 CPUs & 16 DIMMs	CPU1: P1-DIMMA1/P1-DIMMB1/P1-DIMMC1/P1-DIMMD1/P1-DIMME1/P1-DIMMF1/P1-DIMMG1/P1-DIMMH1CPU2: P2-DIMMA1/P2-DIMMB1/P2-DIMMC1/P2-DIMMD1/P2-DIMME1/P2-DIMMF1/P2-DIMMG1/P2-DIMMH1

DDR5 Memory Support for the 4th Gen Intel Xeon Scalable Processors
Type	Ranks Per DIMM & Data Width (Stack)	DIMM Capacity (GB)
			1DPC (Note)
		16GB 1.1V
RDIMM	SRx8 (RC D) 16GB		4800
	SRx4 (RC C) 32GB
	SRx4 (RC F) 9x4 32GB
	DRx8 (RC E) 32GB
	DRx4 (RC A)) 64GB
	DRx4 (RC B) 9x4 64GB
RDIMM 3DS (4R/8R) x4 (RC A)		2H-128GB4H-256GB
LRDIMM/LRDIMM-3DS N/A	N/A Not Supported

Note: 1DPC (1 DIMM per Channel) applies to 1 SPC (Socket Per Channel) or 2 SPC (Soeckets Per Channel) implementation.

Compatible and Incompatible Type in a Channel and a System
DIMM Type RDIMM	RDIMM 3DS 9x4 RDIMM
RDIMM Compatible	Incompatible Incompatible
RDIMM 3DS Incompatible	Compatible Incompatible
9x4 RDIMM Incompatible	Incompatible Compatible

DIMM Installation

1. Insert the desired number of DIMMs into the memory slots based on the recommended DIMM population tables in the previous section. Locate DIMM memory slots on the motherboard as shown on the right.

2. Push the release tabs outwards on both ends of the DIMM slot to unlock it.

Rlease Tabs

1. Align the key of the memory module with the receptive point on the memory slot.

2. Align the notches on both ends of the module against the receptive points on the ends of the slot.

3. Push both ends of the module straight down into the slot until the module snaps into place.

4. Press the release tabs to the lock positions to secure the memory module into the slot.

DIMM Removal

Press both release tabs on the ends of the memory module to unlock it. Once the memory module has been loosened, remove it from the memory slot.

Floor plan diagram of a computer room with labeled rooms and structural elements

Key Receptive Point

Notch Release Tab

Push both ends straight down into the memory slot.

Diagram of a mechanical assembly with blue arrows indicating motion or force direction (no text or symbols)

Warning! Please do not use excessive force when pressing the release tabs on the ends of the DIMM socket to avoid causing any damage to the memory module or the DIMM socket. Please handle memory modules with care. Carefully follow all the instructions given on page 1 of this chapter to avoid ESD-related damages done to your memory modules or components.

3.6 Motherboard Battery

The motherboard uses non-volatile memory to retain system information when system power is removed. This memory is powered by a lithium battery residing on the motherboard.

Replacing the Battery

Begin by removing the node from the system.

1. Push aside the small clamp that covers the edge of the battery. When the battery is released, lift it out of the holder.
2. To insert a new battery, slide one edge under the lip of the holder with the positive (+) side facing up. Then push the other side down until the clamp snaps over it.

Note: Handle used batteries carefully. Do not damage the battery in any way; a damaged battery may release hazardous materials into the environment. Do not discard a used battery in the garbage or a public landfill. Please comply with the regulations set up by your local hazardous waste management agency to dispose of your used battery properly.

LITHIUM BATTERY BATTERY HOLDER

Figure 3-3. Installing the Onboard Battery

Warning: There is a danger of explosion if the onboard battery is installed upside down (which reverses its polarities). This battery must be replaced only with the same or an equivalent type recommended by the manufacturer (BR2032).

3.7 Storage Drives

The CSE-217BQ2 chassis supports 24 hot-swap 2.5" solid-state drives (SSDs). Each node controls six drives. All the drives in the SYS-221BT-HNR support PCIe 5.0. In the other four systems, the first two drives of each node are PCIe 5.0 and the remaining four drives are PCIe 4.0.

• HNR: 24 PCIe 5.0 NVMe drives
• HNTR: 24 hybrid SATA3/NVMe drives
• HNC8R: 24 hybrid SAS3/NVMe drives (SAS supported via onboard Broadcom 3808)
• HNC9R: 24 hybrid SAS3/NVMe drives (SAS supported via Broadcom 3908 AOC)

Drive Carriers

Each drive carrier has two LED indicators: an activity indicator and a status indicator. For RAID configurations using a controller, the meaning of the status indicator is described in the table below. For OS RAID or non-RAID configurations, some LED indications are not supported, such as hot spare.

Drive Carrier LED Indicator
	Color Blinking Pattern Behavior for Device
Activity LED	Blue Solid On	SAS/NVMe drive installed
	Blue Blinking	I/O activity
Status LED	Red Solid On	Failure of drive with RSTe support
	Red Blinking	at 1 Hz Rebuild drive with RSTe support
	Red	Blinking with two blinks and one stop at 1 Hz	Hot spare for drive with RSTe support (not supported in VMD mode)
	Red On for five seconds, then off Power on for drive with RSTe support
	Red Blinking	at 4 Hz Identify drive with RSTe support
	Green Solid On		Safe to remove NVMe device (not supported in VMD mode)
	Amber	Blinking at 1 Hz	Attention state---do not remove NVMe device (not supported in VMD mode)

Note: Enterprise level hard disk drives are recommended for use in Supermicro chassis and servers. For information on recommended HDDs, see the Supermicro website, http://www.supermicro.com/products/nfo/files/storage/SBB-HDDCompList.pdf.

Drive Configuration

The CSE-217BQ2 chassis contains four separate computing node drawers, each with its own motherboard. Each node controls a set of six drives. If a node drawer is pulled out of the chassis, the drives associated with that node will power down.

Node Drawer Locations
Node B controls drives B1, B2, B3, B4, B5, and B6	Node D controls drives D1, D2, D3, D4, D5, and D6
Node A controls drives A1, A2, A3, A4, A5, and A6	Node C controls drives C1, C2, C3, C4, C5, and C6

Node A Node B Node C Node D A B C D

Figure 3-4. Storage Drives and the Corresponding Nodes

Installing Drives

Removing Drive Carriers from the Chassis

1. Press the release button on the drive carrier. This extends the drive carrier handle.
2. Use the handle to pull the carrier out of the chassis (Figure 3-5).
3. Remove the dummy drive from the carrier (Figure 3-6).

Caution: Except for short periods of time (swapping drives), do not operate the server with the drive carriers removed from the bays, regardless of how many drives are installed, for proper airflow.

Technical diagram showing a server rack with labeled components and green directional arrows indicating rotation or assembly.

Figure 3-5. Removing a Drive Carrier

Technical line drawing of an electronic device chassis with labeled ports and connectors (no text or symbols present)

Figure 3-6. Removing a Dummy Drive from the Drive Carrier

Installing a Drive

1. Install a new drive into the carrier with the printed circuit board side facing down so that the mounting holes in the drive align with those in the carrier.

2. Secure the hard drive into the carrier with the screws.

Technical line drawing of a device chassis with a rectangular box and internal components (no text or symbols)

Figure 3-7. Installing the Hard Drive

1. Insert the drive and carrier into its bay vertically, keeping the carrier oriented so that the release button is on the bottom. When the carrier reaches the rear of the bay, the release handle starts to retract.

2. Push the upper part of the drive carrier handle until it clicks into the locked position.

Hot-Swap for NVMe Drives

Supermicro servers support NVMe surprise hot-swap. For even better data security, NVMe orderly hot-swap is recommended. NVMe drives can be ejected and replaced remotely using BMC.

Note: If you are using VROC, see the Hot-Swap Drives section in Chapter 6 instead.

Ejecting a Drive

1. BMC > System > Storage Monitoring > Physical View
2. Select Device, Group and Slot, and click Eject. After ejecting, the drive Status LED indicator turns green.
3. Remove the drive.

Note that Device and Group are categorized by the CPLD design architecture. The SYS-221BT-H Series server has one Device and one Group.

Slot is the slot number on which the NVMe drives are mounted.

Product Dashboard System Component Info Health Event Log Storage Monitoring Configuration Remote Control Maintenance Overview Physical View Logical View Controller Physical View Blank Unblink Eject Insert Blotf LED Status Supported Actions Disk Info#t Capacity Link Speed Connected Logical Drive Connected Controller 0.0 SAMSUNG MZCLW8001MLF- 00003 N/A 8.0 GTIs- N/A N/A White Device0 0.1 SAMSUNG MZCLW8001MLF- 00003 N/A 8.0 GTIs- N/A N/A White Device0

Figure 3-8. BMC Screenshot

Replacing the Drive

1. Insert the replacement drive.
2. BMC > System > Storage Monitoring > Physical View
3. Select Device, Group and slot and click Insert. The drive Status LED indicator flashes red, then turns off. The Activity LED turns blue.

Checking the Temperature of an NVMe Drive

There are two ways to check using BMC.

Checking a Drive

• BMC > Server Health > NVMe SSD – Shows the temperatures of all NVMe drives.
• BMC > Server Health > Sensor Reading > NVME_SSD – Shows the single highest temperature among all the NVMe drives.

3.8 System Cooling

Fans

Fan speed is controlled by a system temperature setting in BMC. If a fan fails, the remaining fans will ramp up to full speed. The system can continue to run with a failed fan. Replace any failed fan at your earliest convenience with the same type and model.

Two Fans Cooling Node A and B Two Fans Cooling Node C and D

Figure 3-9. System Fan Placement

Changing a System Fan

1. Determine which fan is failing. If possible, use BMC. If not, remove the chassis cover while the power is on, and examine the fans to determine which one has failed.
2. Remove power from the system as described in Section 3.1.
3. Remove the fan cable from the backplane for the failed fan and the adjacent fan.
4. Lift the fan housing up and out of the chassis.
5. Push the fan up from the bottom and out of the top of the housing.

HDTV HDTV

Figure 3-10. Replacing a System Fan in the Fan Housing

1. Place the replacement fan into the vacant space in the housing while making sure the arrows on the top of the fan (indicating air direction) point in the same direction as the arrows on the other fans.
2. Put the fan housing back into the chassis and reconnect the cable.
3. Power on the system to confirm that the fan is working properly before replacing the chassis cover.

Installing the Air Shroud

The system requires air shrouds for each node to maximize airflow efficiency.

Installing the Air Shrouds

The motherboard, any expansion cards, and all components must be installed in the node tray. Place the air shroud as pictured below.

Technical line drawing of a server rack with internal components and ventilation ducts (no text or symbols)

Figure 3-11. Installing the Air Shroud

3.9 Power Supply

The chassis features redundant power supplies. The power modules can be changed without powering down the system. New units can be ordered directly from Supermicro or authorized distributors.

These power supplies are auto-switching capable. This feature enables them to automatically sense the input voltage and operate at a 100-120 V or 180-240 V. An amber light will be illuminated on the power supply when the power is off. An illuminated green light indicates that the power supply is operating.

Replacing the Power Supply

1. Unplug the AC cord from the module to be replaced.
2. Push the release tab on the back of the power supply as illustrated.

2 Release Tab 3

Figure 3-12. Power Supply Release Tab

1. Pull the power supply out using the handle provided.
2. Replace the failed power module with the same model.
3. Push the new power supply module into the power bay until it clicks.
4. Plug the AC power cord back into the module.

3.10 AIOM Card

The system supports various AIOM cards to provide networking functionality. See the Supermicro product page for details about the different AIOM cards.

Removing the AIOM Card

1. Press the release tab and loosen the thumbscrew on the AIOM card.

2. Grasp the release tab and the thumbscrew and pull the AIOM out of the node tray.

Thumbscrew Release Tab

Figure 3-13. AIOM Card Node Position

Installing the AIOM Card

1. If necessary, loosen the thumbscrew and remove the shield

2. Insert the card into the node until it is connected to the motherboard slot.

3. Tighten the thumbscrew.

3.11 Expansion Cards

The SYS-221BT-H Series supports two low-profile, half-length x16 expansion cards in each node. Riser cards are used to mount the expansion cards.

Left Expansion Slot Right Expansion Slot

Figure 3-15. PCI Slot Configuration

Expansion Slot Locations (per node)
Expansion Slot	Description (all PCIe 5.0)	Riser Card Model
Left x16 low-profile slot (CPU2)		SCC-P2HM2G4-B1 (Default)SCC-A2NM2241G3-B1 (Optional)	SYS-221BT-HNRSYS-221BT-HNTRSYS-221BT-HNC8RSYS-221BT-HNC9R
Right x16 low-profile slot (CPU1) RSC-PR-6G5			SYS-221BT-HNRSYS-221BT-HNTRSYS-221BT-HNC8R

Populating Right Expansion Slot (SYS-221BT-HNC9R)

1. If necessary, power down the node and remove it from the chassis.
2. Open the latch at the center of the node and remove the I/O slot shield by sliding it towards the center.

Latches PCI Slot Shield

Figure 3-15. PCI Slot Shield and Latches

1. Attach the riser card to the riser card bracket and insert the expansion card into the riser card slot.

Technical line drawing of a server rack with internal components and mounting hardware (no text or symbols)

1. Align the assembly with slots on the motherboard and the PCI slot shield at the node rear. Insert the assembly into the motherboard.
2. Connect cables. See Section 3.14 Cable Routing Diagrams, Figure 3-21.
3. Close the latch, reinsert the node into the chassis, and power up the system.

Populating Left Expansion Slot (All Models)

1. If necessary, power down the node and remove it from the chassis.
2. Open the latch at the center of the node and remove the I/O slot shield by sliding it towards the center.

Latches PCI Slot Shield

Figure 3-15. PCI Slot Shield and Latches

1. If necessary, install M.2 SSDs (not shown) on SCC-P2HM2G4-B1 or SCC-A2NM2241G3-B1.
2. Align the assembly with slots on the motherboard and the PCI slot shield at the node rear. Insert the assembly into the motherboard.
3. Connect cables, if needed.
4. Close the latch, reinsert the node into the chassis, and power up the system.

M.2 SSD Riser Slot Bracket Bracket Latch PCI Slot

Figure 3-16. Installing Expansion Card

1. Check the PCIe bifurcation and the NVMe Firmware Source settings for the M.2 SSDs.

2. BIOS Setup -> Advanced -> Chipset Configuration -> North Bridge -> IIO Configuration -> CPU Configuration -> IOU Setting -> x4x4x4x4

3. BIOS Setup -> Advanced -> PCIe/PCI/PnP Configuration -> NVMe Firmware Source -> AMI Native Support

3.12 Backplane

The backplane (BPN-NVME5-217BHQ-S6) supports 24 SATA or NVMe drives. Four connectors are located at the back side of the backplane, one connected to each node. A cable linking the NearStack HD connector on the adapter to a high speed connector (P1 NS1) on the motherboard's PCB provides PCIe 5.0 support. For 24 SAS drive support, additional SAS adapters and cables are required.

Green printed circuit board with multiple slots and connectors (no readable text or symbols)

Figure 3-17. SATA/NVMe Backplane (Side Facing Chassis Rear)

3.13 Storage Adapters

The storage adapter (SCC-P6N6SGH-B1) connects directly to the backplane and provides a SlimSAS x8 connector. It is connected to the motherboard on SXB1 and powered by four 6-pin power connectors (JPWR1 to 4). The SCC-P6N6SGH-B1 adapter is included for the SYS-221BT-HNTR, SYS-221BT-HNC8R, and SYS-221BT-HNC9R.

Close-up of a green printed circuit board with electronic components and connectors (no readable text or symbols)

Figure 3-18. Storage Adapter (SCC-P6N6SGH-B1)

The storage adapter (SCC-P6NG5-B1) connects directly to the backplane and provides a NearStack HD x8 connector. It is connected to the motherboard on SXB1 and powered by four 6-pin power connectors (JPWR1 to 4). The SCC-P6NG5-B1 is included for the SYS-221BT-HNR.

Close-up of a green printed circuit board (PCB) with visible components and connectors, no readable text or symbols beyond product labels.

Figure 3-19. Storage Adapter (SCC-P6NG5-B1)

3.14 Cable Routing Diagrams

Refer to the storage cable diagrams in the next few pages. When disconnecting cables to add or replace components, refer to the diagrams so you can reroute them in the same manner.

All Flash NVMe Gen5

NVMe connections from the backplane to the motherboard.

NVMe PCIe 5.0 Cable CBL-NSH5-1326

Online Cable Matrix

Internal view of a computer motherboard with labeled components and warning labels in Chinese

Figure 3-19. SYS-221BT-HNR Cable Routing Diagram

NVMe and SATA

NVMe connections from the backplane to the motherboard. SATA connections from the backplane to the motherboard.

NVMe PCIe 5.0 Cable CBL-NSH5-1326

SATA Cable CBL-SAST-1243-100

Online Cable Matrix

SINOC BLOOD IN TPO IF WIREE E-PTA FILLION EXCELING 1 MOZZU SERVICE TO COMP TPO VOUT BIV TELLE 2 MODERN ELECTRONIC EXP CORN BY INFO 1 ENGINE ALL ITEMS ARE IN ALCLOSED POSITION 2 INSTALL PM 3 DRIVE ALL MIND TO CLORECT POSITION 4 YOUTRUN ON NOT TERMINATE 20 MINI INSTALL IN RESISTOR ORDER

Figure 3-20. SYS-221BT-HNTR Cable Routing Diagram

NVMe and SAS via onboard 3808

NVMe connections from the backplane to the motherboard. SAS connections from the backplane to the SCC-B8SB80-B1. The SCC-B8SB80-B1 connects to the motherboard SXB4. The SXB3 is blocked by the SCC-B8SB80-B1 and is unusable.

NVMe PCIe 5.0 Cable CBL-NSH5-1326

SAS Cable CBL-SAST-1221-100

Online Cable Matrix

Internal view of a computer motherboard with labeled components and annotations in Chinese

Figure 3-21. SYS-221BT-HNC8R Cable Routing Diagram

NVMe and SAS via 3908 AOC

NVMe connections from the backplane to the motherboard. SAS connections from the backplane to the AOC-S3908L-H8iR-16DD. The riser card connects to the motherboard Slot 1.

NVMe PCIe 5.0 Cable CBL-NSH5-1326

Hardware RAID Cable CBL-SAST-1234F-100

Online Cable Matrix

3908 AOC not shown 1. ENGINE ALL ITEMS ARE A VALIDED POSITION 2. INSTALL PMR 3. WORK ALL ITEMS TO LOCKED POSITION 4. SHIFTEN EACH PUT TO CALLY SIMAN AC 5. INSTALL A/RISHER ORDER *3908 AOC not shown

Figure 3-22. SYS-221BT-HNC9R Cable Routing Diagram

Chapter 4

Motherboard Connections

This section describes the connections on the motherboard and provides pinout definitions.

Note that depending on how the system is configured, not all connections are required.

The LEDs on the motherboard are also described here. A motherboard layout indicating component locations may be found in Chapter 1. More detail can be found in the

Motherboard Manual.

Please review the Safety Precautions in Appendix A before installing or removing components.

4.1 Power Connections

Power Supply Connectors

Four 6-pin power connectors, located at JPWR1 \~JPWR4, provide main power to your system. All these power connectors meet the ATX SSI EPS 12V specification and must be connected to your power supply to provide adequate power to your system.

12V 6-pin Power Pin Definitions

Pin# Definition

1 - 3 Ground

4 - 6 +12V

4.2 Headers and Connectors

Fan Headers

A 4-pin fan header (FAN3) is used for your system cooling fan. Fan speed control is supported by Thermal Management via the BMC 2.0 interface.

VROC RAID Key Header

A VROC RAID Key header is located at JRK1 on the motherboard. Install a VROC RAID Key on JRK1 for NVMe RAID support as shown in the illustration below.

Intel VROC KeyPin Definitions
Pin#	Definition
1	Ground
2	3.3V Standby
3	Ground
4	PCH RAID Key

VROC Key VROC Key Header (JRK1)

Note: The graphics contained in this user's manual are for illustration only. The components installed in your system may or may not look exactly the same as the graphics shown in the manual.

CPLD Header

A CPLD header is at JCPLD1 on the motherboard. This header supports a Complex Programmable Logic device (CPLD) for system firmware enhancement.

TPM/Port 80 Header

The JTPM1 header is used to connect a Trusted Platform Module (TPM)/Port 80, which is available from Supermicro (optional). A TPM/Port 80 connector is a security device that supports encryption and authentication in hard drives. It allows the motherboard to deny access if the TPM associated with the hard drive is not installed in the system. Please go to the following link for more information on the TPM: http://www.supermicro.com/manuals/other/TPM.pdf.

Trusted Platform Module Header Pin Definitions
Pin# Definition Pin# Definition
1 +3.3V 2 SPI_CS#
3 RESET# 4 SPI_MISO
5 SPI_CLK 6 GND
7 SPI_MOSI 8 NC
9 +3.3V Stdby 10 SPI_IRQ#

NC-SI Connector

The NC-SI (Network Controller Sideband Interface) connector is located at (JNCSI1). This connector is used to connect a Network Interface Card (NIC) to the motherboard to allow the onboard BMC (Baseboard Controller) to communicate with a network.

Note: For detailed instructions on how to configure Network Interface Card (NIC) settings, please refer to the Network Interface Card Configuration User's Guide posted on the web page under the link: http://www.supermicro.com/support/manuals/.

P1 NS1/P2 NS2/P2 NS3 Connectors

Three PCIe 5.0 x8 connectors are located at P1 NS1, P2 NS2, and P2 NS3 headers. P1 NS1 PCIe 5.0 x8 connector (NS1) is supported by CPU1, while P2 NS2 5.0 x8 (NS2) and P2 NS3 5.0 x8 (NS3) are supported by CPU2. Refer to the X13DET-B for the locations of P1 NS1, P2 NS2, and P2 NS3 slots.

PCIe 5.0 x8 NVMe M.2 Slot

A PCIe 5.0 x8, located at SXB5, supports two NVMe M.2 connections in the 2280 or 22110 form factor, which depends on the type of the riser card used on the slot. M.2 allows for a variety of card sizes, increased functionality, and spatial efficiency. Refer to the layout below for the location of the SXB5 slot.

Note: The SXB5 slot supports the SCC-P2HM2G4-B1 and the SCC-A2NM2241G3-B1. The SCC-P2HM2G4-B1 supports two NVMe M.2 devices (up to Gen 4) in the 22110 form factor, and the SCC-A2NM2241G3-B1 supports two NVMe M.2 devices (up to Gen 3) in the 2280 form factor.

SATA 3.0 Ports 0\~7/8\~11

Two SATA 3.0 headers, located at (SATA1/SATA2), supports 12 SATA 3.0 connections (SATA 0\~7 and SATA 8\~11). These SATA 3.0 ports are supported by the Intel C741 Emmitsburg PCH chipset. Connecting proper SATA cables to SATA1 and SATA2 to use SATA 3.0 connections.

AIOM PCIe 5.0 x16 Slot (JAIOM)

A PCIe 5.0 x16 slot, supported by CPU1, is located at JAIOM. This slot is used for an Advanced I/O module or riser card. Refer to the motherboard for the location of the JAIOM slot.

PCIe 5.0 x8 NVMe M.2 Slot

A PCIe 5.0 x8, located at SXB5, supports two NVMe M.2 connections in the 2280 or 22110 form factor, which depends on the type of the riser card used on the slot. M.2 allows for a variety of card sizes, increased functionality, and spatial efficiency. Refer to the X13DET-B for the location of the SXB5 slot.

Note: The SXB5 slot supports the SCC-P2HM2G4-B1 and the SCC-A2NM2241G3-B1. The SCC-P2HM2G4-B1 supports two NVMe M.2 devices (up to Gen 4) in the 22110 form factor, and the SCC-A2NM2241G3-B1 supports two NVMe M.2 devices (up to Gen 3) in the 2280 form factor.

4.3 Input/Output Ports

VGA Connector

A VGA connection is located at JVGA1 on the rear I/O panel. This VGA connector provides analog interface between the computer and the video displays.

COM Port

A COM (communication) port that supports serial link interface is located on the rear side of the motherboard. Refer to the X13DET-B for the location of COM1.

BMC LAN

A dedicated BMC LAN (BMC LAN) is located on the rear I/O panel. The dedicated BMC LAN provides LAN support for the BMC (Baseboard Management Controller). Connect an RJ45 cable to JUSBRJ45 on the I/O rear panel for BMC LAN support.

Universal Serial Bus (USB) Ports and Headers

There are two USB 3.1 ports (USB0/1), located at JUSB1, on the rear I/O panel. These USB ports and headers can be used for USB support via USB cables (not included).

Rear I/O Panel USB 0/1 (3.1)Pin Definitions
Pin# Definition Pin# Definition
A1 VBUS B1 Power
A2 D- B2 USB_N
A3 D+ B3 USB_P
A4 GND B4 GND
A5 Stda_SSRX- B5 USB3_RN
A6 Stda_SSRX+ B6 USB3_RP
A7 GND B7 GND
A8 Stda_SSTX- B8 USB3_TN
A9 Stda_SSTX+ B9 USB3_TP

UID/BMC Reset Switch and UID\_LED/BMC LED Indicator

A UID LED/BMC Reset switch (JUIDB1) is located on the rear I/O panel of the motherboard. This switch has dual functions. It can function as a unit identifier (UID) to identify a system unit that is in need of service, and it can also be used for BMC reset.

To achieve these purposes, JUIDB1 works in conjunction with the UID_LED1/LEDBMC LED indicator located on the rear I/O panel and the UID/BMC LED indicator on the front panel. When functioning as a UID LED switch, JUIDB1 will turn on/off both blue UID indicators located on the front and the rear panels when the user presses this switch on/off.

When functioning as a BMC reset switch, JUIDB1 will reset the BMC settings to the manufacturer defaults when the user presses and holds the switch for 12 seconds, and it will trigger a cold reboot when the user presses and holds it for 6 seconds.

Refer to the BMC User's Guide posted on our website at http://www.supermicro.com for more information.

UID/BMC Reset Switch (JUIDB1)Features & Settings
When Used as a UID LED Switch When Used as a	BMC Reset Switch
UID_LED1/LEDBM acting as a rear UID LED indicator	UID_LED1/LEDBMC acting as a BMC Reset LED indicator.
UID_LED1/LEDBMC: Blue: Unit in need of services is identified	UID_LED1/LEDBMC: Green Blinking: BMC (BMC Heartbeat) Normal
	BMC Reset: Press & hold the switch (JUIDB1) 6 seconds	UID_LED1/LEDBMC: Solid green during reboot
Press the switch (JUIDB1) on/off to turn on/off both rear UID LED (located at UID_LED1/LEDBMC LED) and front UID LED (located on the front control panel) for UID indications		Triggering a cold reboot; the UID_LED1/LEDBMC indicator: Solid green on during cold reboot
	BMC Reset: Press & hold the switch (JUIDB1) 12 seconds	UID_LED1/LEDBMC: Solid green during BMC reset
		BMC: Reset to the manufacturer default; UID_LED1/LEDBMC: Solid on during BMC Reset

UID/BMC Reset Switch (JUIDB1) Pin Definitions
Pin# Definition
1 Ground
2 Ground
3 Button In
4 Button In

4.4 Jumpers

How Jumpers Work

To modify the operation of the motherboard, jumpers can be used to choose between optional settings. Jumpers create shorts between two pins to change the function of the connector. Pin 1 is identified with a square solder pad on the printed circuit board. Refer to the diagram below for an example of jumping pins 1 and 2. Refer to the motherboard layout page for jumper locations.

Note: On two-pin jumpers, "Closed" means the jumper is on, and "Open" means the jumper is off the pins.

graph TD
    A["Connector Pins"] --> B["Top View"]
    B --> C["Connector Pins with a Jumper installed on Pins 1 and 2"]

CMOS Clear

JBT1 is used to clear CMOS, which will also clear any passwords. Instead of pins, this jumper consists of contact pads to prevent accidentally clearing the contents of CMOS.

To Clear CMOS

1. First power down the system and unplug the power cord(s).
2. Remove the cover of the chassis to access the motherboard and remove the battery from the motherboard.
3. Short the CMOS pads, JBT1, with a metal object such as a small screwdriver for at least four seconds.
4. Remove the screwdriver (or shorting device).
5. Re-install the motherboard battery.
6. Replace the cover, reconnect the power cord(s), and power on the system.

Note 1: Clearing CMOS will also clear all passwords.

Note 2: Do not use the PW_ON connector to clear CMOS.

ME Recovery

JPME1 is used for ME Firmware Recovery mode, which will limit system resource for essential function use only without putting restrictions on power use. In the single operation mode, online upgrade will be available via Recovery mode. See the table below for pin definitions.

ME Recovery (JPME1)Jumper Settings
Jumper Setting Definition
Pins 1-2 Normal	(Default)
Pins 2-3 ME Recovery

ME (Manufacture Mode) Select

Use Jumper JPME2 to set ME Mode for your system to enhance your system performance. See the table below for pin definitions.

ME Select (JPME2)Jumper Settings

Jumper Setting Definition

Pins 1-2 ME Mode Enabled (Default)

Pins 2-3 ME Mode Disabled

4.5 LED Indicators

BMC LAN LEDs

A dedicated BMC LAN connection, located on the rear I/O panel, provides Ethernet network connection via the BMC (Baseboard Management Control). The LED on the right indicates activity, and the LED on the left indicates the speed of the connection. Refer to the table below for more information.

BMC LAN LEDs
Color/State Definition
Link (Left)	Green: Solid	100 Mbps
	Amber: Solid	1Gbps
Activity (Right) Amber	Blinking Active

BMC LAN

Activity LEDLink LED

CPLD Heartbeat LED

A Complex Programmable Heartbeat LED indicator is located at LE1. When this LED is on, CPLD is active. Refer to the layout below for the location of LE1.

CPLD Heartbeat LED (LE1)LED State
State Definition
On CPLD	Active (Normal)

Power Status LED

The Power Status LED, located at LE6, indicates the status of onboard power. When this LED turns red, it indicates a power failure. When this LED is green, onboard power is on. When the LED turns amber, the standby power is on. Refer to the layout below for the location of LE6.

Power Status LED (LE6)LED State
Color Definition
Green Power On
Amber Standby Power On
Red	Power Failure

Chapter 5

Software

After the hardware has been installed, you can install the Operating System (OS), configure RAID settings and install the drivers.

5.1 Microsoft Windows OS Installation

If you will be using RAID, you must configure RAID settings before installing the Windows OS and the RAID driver. Refer to the RAID Configuration User Guides posted on our website at www.supermicro.com/support/manuals.

Installing the OS

1. Create a method to access the MS Windows installation ISO file. That might be a USB flash or media drive, or the BMC KVM console.
2. Retrieve the proper RST/RSTe driver. Go to the Supermicro web page for your motherboard and click on "Download the Latest Drivers and Utilities", select the proper driver, and copy it to a USB flash drive. For Driver Installation instructions see Section 5.2.
3. Boot from a bootable device with Windows OS installation. You can see a bootable device list by pressing F11 during the system startup.

Please select boot device: ATEN Virtual CDROM YSOJ → IPMI virtual drive (Legacy) ASUS SDRW-08D2S-U F601 → USB DVD device (Legacy) USB FLASH DRIVE PMAP → USB flash drive with OS installation (Legacy) IBA 40-10G Slot 1900 v1060 → PXE boot (Legacy) UEFI: ATEN Virtual CDROM YSOJ → IPMI virtual drive (UEFI) UEFI: ASUS SDRW-08D2S-U F601 → USB DVD device (UEFI) UEFI: Built-in EFI Shell Enter Setup ↑ and ↓ to move selection ENTER to select boot device ESC to boot using defaults

Figure 5-1. Select Boot Device

1. During Windows Setup, continue to the dialog where you select the drives on which to install Windows. If the disk you want to use is not listed, click on "Load driver" link at the bottom left corner.

Where do you want to install Windows? Name Total size Free space Type Refresh Delete Format New Load driver Extend We couldn't find any drives. To get a storage driver, click Load driver. Next

Figure 5-2. Load Driver Link

To load the driver, browse the USB flash drive for the proper driver files.

• For RAID, choose the SATA/sSATA RAID driver indicated then choose the storage drive on which you want to install it.

• For non-RAID, choose the SATA/sSATA AHCI driver indicated then choose the storage drive on which you want to install it.

• Once all devices are specified, continue with the installation.

• After the Windows OS installation has completed, the system will automatically reboot multiple times.

5.2 Driver Installation

The Supermicro website contains drivers and utilities for your system at https://www.supermicro.com/wdl/driver. Some of these must be installed, such as the chipset driver.

After accessing the website, go into the CDR_Images (in the parent directory of the above link) and locate the ISO file for your motherboard. Download this file to a USB flash or media drive. (You may also use a utility to extract the ISO file if preferred.)

Another option is to go to the Supermicro website at http://www.supermicro.com/products/. Find the product page for your motherboard, and "Download the Latest Drivers and Utilities". Insert the flash drive or disk and the screenshot shown below should appear.

SUPERMICRO X13DET-B Motherboard Drivers & Tools (Win11) Intel® C741 Chipset X13DET-B SUPERMICRO Computer Inc. Intel Chipset INF files Microsoft .Net Framework 4.8(Optional) ASPEED Graphics Driver Intel Virtual RAID on CPU Intel PRO Network Connections Drivers Microsoft .Net SDK SUPERMICRO SuperDoctor 5 Build driver diskettes and manuals Browse CD Auto Start Up Next Time For more information, please visit SUPERMICRO's web site.

Figure 5-3. Driver & Tool Installation Screen

Note: Click the icons showing a hand writing on paper to view the readme files for each item. Click the computer icons to the right of these items to install each item (from top to the bottom) one at a time. After installing each item, you must re-boot the system before moving on to the next item on the list. The bottom icon with a CD on it allows you to view the entire contents.

5.3 SuperDoctor® 5

The Supermicro SuperDoctor 5 is a program that functions in a command-line or web-based interface for Windows and Linux operating systems. The program monitors such system health information as CPU temperature, system voltages, system power consumption, fan speed, and provides alerts via email or Simple Network Management Protocol (SNMP).

SuperDoctor 5 comes in local and remote management versions and can be used with Nagios to maximize your system monitoring needs. With SuperDoctor 5 Management Server (SSM Server), you can remotely control power on/off and reset chassis intrusion for multiple systems with SuperDoctor 5 or BMC. SuperDoctor 5 Management Server monitors HTTP, FTP, and SMTP services to optimize the efficiency of your operation.

SuperDoctor® Manual and Resources

SuperDoctor 5 Health Info System Info Configuration Report Power Control Select Tempsuse: English [admin] Login Motherboard: X8DTU-LN4+ Fan Speed FAN S FAN B Voltage CPU1 Vccr CPU1 DINM +1.5 V -3 V +5VSB +12 V +1.1 V +3.3VCC +3.3VSB VBAT Temperature 100 -210 80 -46 80 -126 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 100 -210 100 -210 80 -46 80 -126 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 100 -210 100 -210 80 -46 80 -126 478.4 87M 78.4 87M 51.2 51.2 51.2 +5VSB +5VSB +5VSB +5VSB +5VSB +5VSB +5VSB +5VSB +5VSB +5VSB +5VSB +5VSB +5VSB +5VSB +5VSB +5VSB +5VSB +5VSB +5VSB +5VSB +5V SB +5V SB +5V SB +5V SB +5V SB +5V SB +5V SB +5V SB +5V SB +5V SB +5V SB +5V SB +5V SB +5V SB +5V SB +5V SB +5V SB +5V SB +5V SB +5V SB +5V SBS +5V SBS +5V SBS +5V SBS +5V SBS +5V SBS +5V SBS +5V SBS +5V SBS +5V SBS +5V SBS +5V SBS +5V SBS +5V SBS +5V SBS +5V SBS +5V SBS +5 VSB +5 VSB +5 VSB +5 VSB +5 VSB +5 VSB +5 VSB +5 VSB +5 VSB +5 VSB +5 VSB

Figure 5-4. SuperDoctor 5 Interface Display Screen (Health Information)

5.4 BMC

The motherboard provides remote access, monitoring and management through the baseboard management controller (BMC) and other management controllers distributed among different system modules. There are several BIOS settings that are related to BMC. For general documentation and information on BMC, visit our website at:

www.supermicro.com/en/solutions/management-software/bmc-resources

BMC ADMIN User Password

For security, each node is assigned a unique default BMC password for the ADMIN user. This can be found on a sticker on the chassis, on each motherboard, or on each node. The stickers also displays the BMC MAC address.

BMC AC1F6BC PWD SUOKJ

Figure 5-5. BMC Password Label

See Chapter 1 for the location of the labels.

Chapter 6

Optional Components

This chapter describes optional system components and installation procedures.

6.1 Optional Parts List

Optional Parts List
Description Part Number
Trusted Platform Module (TPM)	AOM-TPM9670AOM-TPM-9671V
Virtual RAID on CPU (VROC) Hardware Key	AOC-VROCPREMODAOC-VROCSTNMOD
M.2 NVMe HW RAID Controller Card SCC-A2NM2241G3-B1

6.2 TPM Security Module

SPI capable TPM 2.0 (or 1.2) with Infineon 9670 controller, vertical form factor

The JTPM1 header is used to connect a Trusted Platform Module (TPM). A TPM is a security device that supports encryption and authentication in hard drives. It enables the motherboard to deny access if the TPM associated with the hard drive is not installed in the system.

Details and installation procedures are at:

http://www.supermicro.com/manuals/other/TPM.pdf.

• AOM-TPM-9670V
  • AOM-TPM-9671V

6.3 Intel Virtual RAID on CPU (VROC)

Intel® Virtual RAID on CPU (Intel VROC) is an enterprise RAID solution for NVMe SSDs directly attached to Intel Xeon Scalable processors. Intel Volume Management Device (VMD) is an integrated controller inside the CPU PCIe root complex.

• A single processor supports up to 12 NVMe SSDs and up to 6 RAID arrays.
• A dual processor system supports up to 24 NVMe SSDs and 12 RAID arrays.

Stripe sizes are 4K, 8K, 16K, 32K, 64K, 128K.

Requirements and Restrictions

• Intel VROC is only available when the system is configured for UEFI boot mode.
• To enable the mdadm command and support for RSTe, install the patch from

- Linux: https://downloadcenter.intel.com/download/28158/Intel-Virtual-RAID-on-CPU-Intel-VROC-and-Intel-Rapid-Storage-Technology-enterprise-Intel-RSTe-Driver-for-Linux-

- Windows: https://downloadcenter.intel.com/download/28108/Intel-Virtual-RAID-on-CPU-Intel-VROC-and-Intel-Rapid-Storage-Technology-enterprise-Intel-RSTe-Driver-for-Windows-

• To enable Intel VROC, a hardware key must be inserted on the motherboard, and the appropriate processor's Virtual Management Devices must be enabled in the BIOS setup.
• It is possible to enable Intel VROC without a hardware key installed, but only RAID0 will be enabled.
• Intel VROC is not compatible with secure boot. This feature must be disabled.
• When creating bootable OS RAID1 devices, you must have both devices on the same CPU, and a VMD on that CPU.
• Spanning drives when creating RAID devices is not recommended due to performance issues, even though it is supported.

Supported SSDs and Operating Systems

To see the latest support information: https://www.intel.com/content/www/us/en/support/articles/000030310/memory-and-storage/ssd-software.html

Additional Information

Additional information is available on the product page for the Supermicro add-on card and the linked manuals.

www.supermicro.com/products/accessories/addon/AOC-VROCxxxMOD.cfm

Hardware Key

The Intel VROC hardware key is a license key that detects the Intel VROC SKU and activates the function accordingly. The key must be plugged into the Supermicro motherboard (connector JRK1). The key options are:

Intel® VROC Keys
VROC Package Description	Part Number Intel MM Number
Standard	RAID 0, 1, 10Supports 3rd party SSDs	AOC-VROCSTNMOD 951605
Premium	RAID 0, 1, 5, 10Supports 3rd party SSDs	AOC-VROCPREMOD 951606

VROC Key Header, JRK1

Figure 6-1. Intel VROC RAID Key and Motherboard Connector JRK1

Configuring Intel VMD

VMD must be enabled on PCIe ports which have NVMe drives attached to them in order for those drives to be added to a VROC RAID configuration. The default BIOS setting for the NVMe Mode Switch is Auto which automatically enables VMD on all installed NVMe drives.

NVMe Mode Switch :

• Auto Enables VMD for all NVMe ports if VROC Key is installed.
  • VMD Enables VMD for all NVMe ports despite the lack of the VROC Key.
• Manual Allows the user to select specific NVMe ports on which to enable VMD.

The NVMe Mode Switch can be viewed or selected at BIOS > Advanced > Chipset Configuration > North Bridge > IIO Configuration > Intel® VMD Technology.

Note: Without a VROC Key, there is no RAID support with the Auto switch. Only RAID 0 is supported with the VMD and Manual switches.

Configuring VMD Manually

The steps for manually configuring VMD on specific NVMe ports in UEFI BIOS are shown below. Example screenshots may differ from your server.

1. Reboot the server and press [DEL] key to access the BIOS options.
2. Switch to Advanced > Chipset Configuration > North Bridge > IIO Configuration > Intel® VMD Technology.
3. Select VMD Mode Switch, then select Manual.

Aptio Setup - AMI Advanced Intel VMD Technology NVMe Mode Switch [Auto] ► Intel VMD for Volume Management Device on Socket 0 ► Intel VMD for Volume Management Device on Socket 1 NVMe Mode Switch Manual VMD Auto Select NVMe Mode, default Auto mode - enable VMD when VROC key presence

Note that Socket 0 contains CPU1; Socket 1 contains CPU2

Figure 6-2. BIOS, Selecting VMD Mode

Note: Consult the X13DET-B Motherboard Manual for more detailed information on setting up Intel VMD.

Caution: VMD must only be enabled on NVMe port resources. If VMD is enabled on other PCIe ports, the functionality of those ports will be impacted. See the table below.

1. Select "Intel VMD for Volume Management Device on" on Socket 0 (CPU1) or Socket 2 (CPU2) to enable VMD for devices under the respective CPU.

Aptio Setup - apt Advanced VAD Config for POH parts Enable/Disable VM0 [Disable] VAD Config for IDU 3 Enable/Disable VM0 [Disable] VAD Config for IDU 1 Enable/Disable VM0 [Disable] VAD Config for IDU 2 Enable/Disable VM0 [Disable] VAD Config for IDU 3 Enable/Disable VM0 [Disable] VAD Config for IDU 4 Enable/Disable VM0 [Disable] VAD Config for IDU 5 Enable/Disable VM0 [Disable] VAD Config for IDU 8 Enable/Disable VM0 [Disable] Enable/Disable VM0 [Disable] Enable/Disable VM0 in this Stack... ++: Select Screen TI: Select Item Enter: Select +/-: Change Opt. F1: General Help F2: Previous Values F3: Optimized Defaults F4: Save & Exit

Aptio Setup - AMI Advanced VMD Config for IDU 0 Enable/Disable VMD [Disable] VMD Config for IDU 1 Enable/Disable VMD [Disable] VMD Config for IDU 2 Enable/Disable VMD [Disable] VMD Config for IDU 3 Enable/Disable VMD [Disable] VMD Config for IDU 4 Enable/Disable VMD [Disable] VMD Config for IDU 5 Enable/Disable VMD [Disable] VMD Config for IDU 6 Enable/Disable VMD [Disable] Enable/Disable VMD [Disable] Enable/Disable VMD [Disable] Enable/Disable VMD [Disable] Enable/Disable VMD [Disable] Enable/Disable VMD [Disable] Enable/Disable VMD [Disable] Enable/Disable VMD [Disable] Enable/Disable VMD [Disable] Enable/Disable VMD [Disable] Enable/Disable VMD [Disable] Enable/Disable VMD [Disable] Enable/Disable VMD [Disable] Enable/Disable WMD [Disable] Enable/Disable WMD [Disable] Enable/Disable WMD [Disable] Enable/Disable WMD [Disable] Enable/Disable WMD [Disable] Enable/Disable WMD [Disable] Enable/Disable WMD [Disable] Enable/Disable WMD [Disable] Enable/Disable WMD [Disable] Enable/Disable WMD [Disable] Enable/Disable WMD [Disable] Enable/Disable WMD [Disable] Enable/Disable WMD [Disable]

Figure 6-3. Intel VMD for Volume Management Device on Socket 0 and Socket 1

1. Choose Enable for "Enable/Disable VMD" for IOU 3 to list the available devices under

Aptio Setup - AMI Advanced VMD Config for IOU 0 Enable/Disable VMD [Disable] VMD Config for IOU 1 Enable/Disable VMD [Disable] VMD Config for IOU 2 Enable/Disable VMD [Disable] VMD Config for IOU 3 Enable/Disable VMD VMD Config for IOU 4 Enable/Disable VMD VMD Config for IOU 5 Enable/Disable VMD [Disable] VMD Config for IOU 6 Enable/Disable VMD [Disable] Enable/Disable VMD in this Stack. +:- Select Screen ↑↓: Select Item Enter: Select +/-: Change Opt.

Figure 6-4. BIOS, Enabling VMD on Socket 1 (CPU2) (Example)

Aptio Setup - AMI Advanced
VMD Config for IOU 0 Enable/Disable VMD [Disable]	Enable/Disable Intel Volume Management Device Technology on specific root port
VMD Config for IOU 1 Enable/Disable VMD [Disable]
VMD Config for IOU 2 Enable/Disable VMD [Disable]
VMD Config for IOU 3 Enable/Disable VMD [Enable] Socket1 IOU3 VMD port E [Disable] Socket1 IOU3 VMD port G [Disable] Hot Plug Capable [Disable]
VMD Config for IOU 4 Enable/Disable VMD [Disable]	+: Select Screen ↑↓: Select Item Enter: Select +/-: Change Opt. F1: General Help F2: Previous Values F3: Optimized Defaults
VMD Config for IOU 5 Enable/Disable VMD [Disable]
VMD Config for IOU 6 Enable/Disable VMD [Disable]

Figure 6-5. BIOS, Enabling VMD on Socket 1 (Example)

1. Enable the NVMe port resource according to table above for the NVMe drives that will be used in a RAID configuration.

Aptio Setup - AMI Advanced
VMD Config for IOU 0 Enable/Disable VMD [Disable]	Enable/Disable Intel Volume Management Device Technology on specific root port
VMD Config for IOU 1 Enable/Disable VMD [Disable]
VMD Config for IOU 2 Enable/Disable VMD [Disable]
VMD Config for IOU 3 Enable/Disable VMD [Enable] Socket1 IOU3 VMD port E [Enable] Socket1 IOU3 VMD port G [Enable] Hot Plug Capable [Disable]
VMD Config for IOU 4 Enable/Disable VMD [Disable]
VMD Config for IOU 5 Enable/Disable VMD [Disable]
VMD Config for IOU 6 Enable/Disable VMD [Disable]
+:-: Select Screen ↑↓: Select Item Enter: Select +/-: Change Opt. F1: General Help F2: Previous Values F3: Optimized Defaults

Figure 6-6. BIOS, Enabling Socket 1 (Example)

1. Choose whether to make the NVMe drives in this IOU Hot Plug Capable by selecting Enabled or Disabled.
2. Repeat steps 4 through 7 for each IOU # on each CPU to enable VMD on the desired NVMe ports.

Aptio Setup - AMI Advanced VMD Config for IOU 0 Enable/Disable VMD [Disable] VMD Config for IOU 1 Enable/Disable VMD [Disable] VMD Config for IOU 2 Enable/Disable VMD [Disable] VMD Config for IOU 3 Enable/Disable VMD [Enable] Socket1 IOU3 VMD port E [Enable] Socket1 IOU3 VMD port G [Enable] Hot Plug Capable [Disable] VMD Config for IOU 4 Enable/Disable VMD [Enable] Socket1 IOU4 VMD port A [Enable] Socket1 IOU4 VMD port C [Enable] Socket1 IOU4 VMD port E [Enable] Socket1 IOU4 VMD port G [Enable] Hot Plug Capable [Disable] VMD Config for IOU 5 Enable/Disable VMD [Disable] Enable/Disable VMD [Disable] Enable/Disable VMD in this Stack. +: Select Screen ↑↓: Select Item Enter: Select +/-: Change Opt. F1: General Help F2: Previous Values F3: Optimized Defaults F4: Save & Exit ESC: Exit VMD Config for IOU 6 Enable/Disable VMD [Disable] F1: General Help F2: Previous Values F3: Optimized Defaults F4: Save & Exit ESC: Exit Version 2.22.1286 Copyright (C) 2022 AMI

Figure 6-7. BIOS, Enabling Socket 1 Completed (Example)
(This example shows SYS-621H-TN12R with 12 NVMe. Other systems will look different.)

1. Press [F4] to save the configuration and reboot the system.

Note: If there is an existing RAID configuration, delete the RAID volume associated with the VMD controller before disabling the controller. Failure to do so may lead to unexpected behavior.

Note: The effects of physically changing or swapping a CPU on the VMD controller have not been thoroughly tested or documented.

Creating NVMe RAID Configurations

1. Open Advanced > Intel(R) Virtual RAID on CPU > All Intel VMD Controllers > Create RAID Volume.

Advanced Create RAID Volume Name: RAID LEVEL: Enable RAID Sp-themed ever VND Controllers: Select Disks: INTEL 100FF2KHOSETZ SWPHAC11040063FORDA, 3576.98GB Port 3/2 CP30 VMD INTEL 100FF2KHOSETZ SWPHAC14730317FORDA, 3576.98GB Port 3/3 CP30 VMD INTEL 100FF2KHOSETZ SWPHAC11040063FORDA, 3576.98GB Port 4/0 CP30 VMD INTEL 100FF2KHOSETZ SWPHAC11040063FORDA, 3576.98GB Port 4/1 CP30 VMD INTEL 100FF2KHOSETZ SWPHAC14720031FORDA, 3576.98GB Port 4/2 CP30 VMD Valued: RAID(Stripe) [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] Enter a unique volume name that does not contain any at the beginning or boccrack and is its characters or less. ++ Select Screen File Select Item Enter Select +/- Change Out. Fit General Help For Previous Values FSI: Optimized Defaults For Save & Exit ESC Exit

Figure 6-8. Created Volume without enabling RAID spanned over VMD controller

Netto Setup - And Advanced Creates RAID Volume Name: RAID Level: Enable RAID Spanned over VMD Controllers: Select Disk(S) DTEL: STOPF2X1087Z DHPHC110400GDP6N, 3576.90GB Port S12 CPU: VHD DTEL: STOPF2X1087Z DHPHC147203TDP6N, 3576.90GB Port S12 CPU: VHD DTEL: STOPF2X1087Z DHPHC10400GDP6N, 3576.90GB Port 412 CPU: VHD DTEL: STOPF2X1087Z DHPHC10400GDP6N, 3576.90GB Port 412 CPU: VHD DTEL: STOPF2X1087Z DHPHC147203TDP6N, 3576.90GB Port 412 CPU: VHD Volume: [DAIDO Stripsel] [Q] [Q] [Q] [Q] [Q] [Q] [Q] X - to Default Disk ←; Select Screen M; Select Item Enter: Select √; Change Up. F1: General Help. P2: Previous Values P3: Optimized Defaults F4: Save & Exit RSC: Exit

Figure 6-9. Created Volume with enabling RAID spanned over VMD controller

1. Set Name.
2. Set RAID Level.
3. If cross-controller RAID is required, select Enable RAID spanned over VMD Controller.
4. Select specific disks for RAID with an [X].

• RAID0: Select at least two [2 - 24] disks
• RAID1: Select only two disks
• RAID5: Select at least three [3 - 24] disks
• RAID10: Select only four disks

1. Select Strip Size (Default 64KB).
2. Select Create Volume.
3. If another RAID is needed, start again at step 1.

Status Indications

An LED indicator on the drive carrier shows the RAID status of the drive.

Drive Carrier Status LED Indicator
Status State (red)
Normal function OFF
Locating 4 Hz blink
Fault Solid ON
Rebuilding 1 Hz Blink

IBPI SFF 8489 Defined Status LED States

Hot-Swap Drives

Intel VMD enables hot-plug and hot-unplug for NVMe SSDs, whether from Intel or other manufacturers. Under vSphere ESXi, several steps are necessary to avoid potential stability issues. See the information at the link [1] below.

Hot-unplug

1. Prevent devices from being re-detected during rescan:

esxcli storage core claiming autoclaim --enabled=false

1. Unmount the VMFS volumes on the device. Check [2] for details.
2. Detach the device. Check [3] for details.
3. Physically remove the device.

Hot-plug

• Physically install the device.

ESXi will automatically discover NVMe SSDs, but a manual scan may be required in some cases.

Related Information Links

[1] https://kb.vmware.com/s/article/2151404
[2] https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.storage.doc/GUID-1B56EF97-F60E-4F21-82A7-8F2A7294604D.html
[3] https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.storage.doc/GUID-F2E75F67-740B-4406-9F0C-A2D99A698F2A.html

Chapter 7

Troubleshooting and Support

7.1 Information Resources

Website

A great deal of information is available on the Supermicro website, supermicro.com.

SUPERMICR Products Solutions Company News Support JumpStart Free Online Access to Cloud Infrastructure Total Solution Test and Validate Your Cloud Infrastructure Online and Deploy On-Prem With Up to 70% Cost Savings

Figure 7-1. Supermicro Website

• Specifications for servers and other hardware are available by the Products option.
• The Support option offers downloads (manuals, BIOS/BMC, drivers, etc.), FAQs, RMA, warranty, and other service extensions.

Direct Links for the SYS-221BT-H Series System

SYS-221BT-HNR specifications page

SYS-221BT-HNTR specifications page

SYS-221BT-HNC8R specifications page

SYS-221BT-HNC9R specifications page

X13DET-B motherboard page for links to the Quick Reference Guide, User Manual, validated storage drives, etc.

SCC-P6NG5-B1

SCC-P6N6SGH-B1

SCC-P2HM2G4-B1

SCC-A2NM2241G3-B1

Direct Links for General Support and Information

Frequently Asked Questions

Add-on card descriptions

TPM User Guide

General Memory Configuration Guide: X13

BMC User Guide

SuperDoctor5 Large Deployment Guide

For validated memory, use our Product Resources page

Product Matrices page for links to tables summarizing specs for systems, motherboards, power supplies, riser cards, add-on cards, etc.

Security Center for recent security notices

Supermicro Phone and Addresses

7.2 Baseboard Management Controller (BMC)

The system supports the Baseboard Management Controller (BMC). BMC is used to provide remote access, monitoring and management. There are several BIOS settings that are related to BMC.

For general documentation and information on BMC, please visit our website at: https://www.supermicro.com/manuals/other/BMC_Users_Guide_X13.pdf

System Disinbees System Component Info Health Event Log Storage Monitoring Configuration + Remote Control Maintenance + System System Storage UIC Control Firmware Update Sensor Readings System Firmware Version 9.10.04 DETA Firmware Build Time 04/10/2020 Redfish Version 1.8.0 BIOS Version BIOS Build Time CPLD Version F9.05.05 BMC MacAddress AC:1F-60.3D 22.0G LAN 1 Mac Address AC:1F-60.CA 22.BA LAN 2 Mac Address AC:1F-60.CA 22.BB LAN 1 Mac Address 6CC47A/D8 SE 7E LAN 2 Mac Address 6CC47A/D8 SE 2F LAN 1 Mac Address AC:1F-80 CF 37.FA LAN 2 Mac Address AC:1F-60 CF 37.FB Host Server Host Name Server IP Address 172.31.56.213 IPv6 Address 1 fe60.0.0.aetf6off.fa3d 3dd3 IPv6 Address 2 IPv6 Address 3 IPv6 Address 4 IPv6 Address 5 IPv6 Address 6 Power Consumption Mix Peak Average Usage Max Peak X:Time(min), Y:Power Consumption (Watt) Remote Console Preview JAVA plug-in HTML6 reload image

Figure 7-2. BMC Sample

7.3 Troubleshooting Procedures

Use the following procedures to troubleshoot your system. If you have followed all of the procedures below and still need assistance, refer to the Technical Support Procedures or Returning Merchandise for Service section(s) in this chapter. Power down the system before changing any non hot-swap hardware components.

General Technique

If you experience unstable operation or get no boot response, try:

1. With power off, remove all but one DIMM and other added components, such as add-on cards, from the motherboard. Make sure the motherboard is not shorted to the chassis.
2. Set all jumpers to their default positions.
3. Power up. If the system boots, check for memory errors and add-on card problems.

No Power

- Check that the power LED on the motherboard is on.

UD_LED J.PME1 LED J.PME1 PWI_LAM USB01 (3.1) VGR LE0 BXC BIOS TEN JHE_XUV LE1 VCC SLOT1 BIOS VCC VCC LE1 VCC SUS PAR CODE ICENS PM CODE SATA1 SATA2-7 PLACMFCe 5.0 X16 PCH VIAO1A SUPER X13DET-B REV:01 UST1

Figure 7-3. Location of the MB Power LED

• Make sure that the power connector is connected to the power supply.
• Check that the motherboard battery still supplies approximately 3 VDC. If it does not, replace it.
• Check that the system input voltage is 100-120 VAC or 180-240 VAC.
• Turn the power switch on and off to test the system

No Video

If the power is on but you have no video, remove all add-on cards and cables.

System Boot Failure

If the system does not display Power-On-Self-Test (POST) or does not respond after the power is turned on, try the following:

- Turn on the system with only one DIMM module installed. If the system boots, check for bad DIMM modules or slots by following the Memory Errors Troubleshooting procedure below.

Memory Errors

• Make sure that the DIMM modules are properly and fully installed.
• Confirm that you are using the correct memory. Also, it is recommended that you use the same memory type and speed for all DIMMs in the system. See Section 3.5 for memory details.
• Check for bad DIMM modules or slots by swapping modules between slots and noting the results.

Losing the System Setup Configuration

• Always replace power supplies with the exact same model that came with the system. A poor quality power supply may cause the system to lose the CMOS setup configuration.
• Check that the motherboard battery still supplies approximately 3 VDC. If it does not, replace it.

If the above steps do not fix the setup configuration problem, contact your vendor for repairs.

When the System Becomes Unstable

If the system becomes unstable during or after OS installation, check the following:

• CPU/BIOS support: Make sure that your CPU is supported and that you have the latest BIOS installed in your system.
• Memory: Make sure that the memory modules are supported. Refer to the product page on our website at www.supermicro.com. Test the modules using memtest86 or a similar utility.

• Storage drives: Make sure that all drives work properly. Replace if necessary.

• System cooling: Check that all heatsink fans and system fans work properly. Check the hardware monitoring settings in the BMC to make sure that the CPU and system temperatures are within the normal range. Also check the Control panel Overheat LED.

• Adequate power supply: Make sure that the power supply provides adequate power to the system. Make sure that all power connectors are connected. Refer to the Supermicro website for the minimum power requirements.
• Proper software support: Make sure that the correct drivers are used.

If the system becomes unstable before or during OS installation, check the following:

• Source of installation: Make sure that the devices used for installation are working properly, including boot devices.
• Cable connection: Check to make sure that all cables are connected and working properly.
• Use the minimum configuration for troubleshooting: Remove all unnecessary components (starting with add-on cards first), and use the minimum configuration (but with a CPU and a memory module installed) to identify the trouble areas.

- Identify a bad component by isolating it. Check and change one component at a time.

• Remove a component in question from the chassis, and test it in isolation. Replace it if necessary.
• Or swap in a new component for the suspect one.
• Or install the possibly defective component into a known good system. If the new system works, the component is likely not the cause or the problem.

7.4 Crash Dump Using BMC

In the event of a processor internal error (IERR) that crashes your system, you may want to provide information to support staff. You can download a crash dump of status information using BMC. The BMC manual is available at https://www.supermicro.com/en/solutions/management-software/bmc-resources.

Check BMC Error Log

1. Access the BMC web interface.

2. Click the Server Health tab, then Event Log to verify an IERR error.

Advanced Settings Filter Sensor-specific Threshold Generator OEM Unspecified Health Event Log Clear Health Event Log Export to Excel Severity Date/Time Sensor Type Categories Description Event Type 2020-10-15 18:37:06 ACPowerOn [OEM | First AC Power on - Assertion] Sensor specific 2020-10-15 18:39:43 System NIC [OEM | Dedicated LAN Link Up - Assertion] Sensor specific 2020-10-15 17:41:37 ACPowerOn [OEM | First AC Power on - Assertion] Sensor specific 2020-10-15 17:41:17 System NIC [OEM | Dedicated LAN Link Up - Assertion] Sensor specific 2020-10-15 17:41:14 System NIC [OEM | Dedicated LAN Link Down - Assertion] Sensor specific 2020-10-15 17:41:12 System NIC [OEM | Dedicated LAN Link Up - Assertion] Sensor specific 1 - 6/6

Figure 7-4. BMC Event Log

In the event of an IERR, the BMC executes a crash dump. You must download the crash dump and save it.

7.5 UEFI BIOS Recovery

Warning: Do not upgrade the BIOS unless your system has a BIOS-related issue. Flashing the wrong BIOS can cause irreparable damage to the system. In no event shall Supermicro be liable for direct, indirect, special, incidental, or consequential damages arising from a BIOS update. If you do update the BIOS, do not shut down or reset the system while the BIOS is updating to avoid possible boot failure.

Overview

The Unified Extensible Firmware Interface (UEFI) provides a software-based interface between the operating system and the platform firmware in the pre-boot environment. The UEFI specification supports an architecture-independent mechanism that will allow the UEFI OS loader stored in an add-on card to boot the system. The UEFI offers clean, hands-off management to a computer during system boot.

Recovering the UEFI BIOS Image

A UEFI BIOS flash chip consists of a recovery BIOS block and a main BIOS block (a main BIOS image). The recovery block contains critical BIOS codes, including memory detection and recovery codes for the user to flash a healthy BIOS image if the original main BIOS image is corrupted. When the system power is turned on, the recovery block codes execute first. Once this process is complete, the main BIOS code will continue with system initialization and the remaining POST (Power-On Self-Test) routines.

Note 1: Follow the BIOS recovery instructions below for BIOS recovery when the main BIOS block crashes.

Note 2: When the BIOS recovery block crashes, you will need to follow the procedures to make a Returned Merchandise Authorization (RMA) request. Also, you may use the Supermicro Update Manager (SUM) Out-of-Band (https://www.supermicro.com.tw/products/nfo/SMS_SUM.cfm) to reflash the BIOS.

Recovering the Main BIOS Block with a USB Device

This feature allows the user to recover the main BIOS image using a USB-attached device without additional utilities used. A USB flash device such as a USB Flash Drive, or a USB CD/DVD ROM/RW device can be used for this purpose. However, a USB Hard Disk drive cannot be used for BIOS recovery at this time.

The file system supported by the recovery block is FAT (including FAT12, FAT16, and FAT32) which is installed on a bootable or non-bootable USB-attached device. However, the BIOS might need several minutes to locate the SUPER.ROM file if the media size becomes too large due to the huge volumes of folders and files stored in the device.

To perform UEFI BIOS recovery using a USB-attached device, follow the instructions below.

1. Using a different machine, copy the "Super.ROM" binary image file into the Root "\" directory of a USB flash or media drive or a writable CD.
   Note 1: If you cannot locate the "Super.ROM" file in your drive disk, visit our website at www.supermicro.com to download the BIOS package. Extract the BIOS binary image into a USB flash device and rename it "Super.ROM" for the BIOS recovery use.
   Note 2: Before recovering the main BIOS image, confirm that the "Super.ROM" binary image file you download is the same version or a close version meant for your motherboard.
2. Insert the USB device that contains the new BIOS image ("Super.ROM") into your USB drive and reset the system when the following screen appears.
3. After locating the healthy BIOS binary image, the system will enter the BIOS Recovery menu as shown below.

SUPERMICO BMC IP:10.132.161.13 PEI--Could Not Find Recovery Image... 07

Note: At this point, you may decide if you want to start the BIOS recovery. If you decide to proceed with BIOS recovery, follow the procedures below.

Option Setup Utility - Copyright (C) 2017 American Megatrends, Inc. Main Advanced Event Log (PVI) Recovery Security Boot Save a Exit Please select blocks you want to update: Boot Black Boot Block Update [Enabled] [Enabled] ▶ Proceed with flash update Get this option to reset WAVM to default values ++ Select Screen T4: Select Item Enter: Select +/- Change Opt. F1: General Help F2: Previous Values F3: Optimized Defaults F4: Save & Exit ESC: Exit Version 2.19.1766 Copyright (C) 2017 American Megatrends, Inc.

1. When the screen as shown above displays, use the arrow keys to select the item "Proceed with flash update" and press the key. You will see the BIOS recovery progress as shown in the screen below.

Note: Do not interrupt the BIOS flashing process until it has completed.

1. After the BIOS recovery process is complete, press any key to reboot the system.
2. Using a different system, extract the BIOS package into a USB flash drive.

Auto Setup Utility - Copyright (C) 2017 American Megatrends, Inc. Recovery WARNING! System firmware is being updated. Keyboard is locked. DO NOT TURN THE POWER OFF!!! Once firmware update is completed press any key to reboot the system Program new data Write new boot block... 178 +: Select Screen T4: Select Item Enter: Select +/-: Change Opt. F1: General Help F2: Previous Values F3: Optimized Defaults F4: Save a Exit DSC: Exit Version 2.19.1266. Copyright (C) 2017 American Megatrends, Inc.

1. Press continuously during system boot to enter the BIOS Setup utility. From the top of the tool bar, select Boot to enter the submenu. From the submenu list, select Boot

Ratio Setup Utility - Copyright (CI 2017 American Mapsfrends, Inc. Recovery WARNING System firmware is being updated. Keyboard is locked... DO NOT TURN THE POWER OFF !!! Once firmware update is completed press any key to reboot the system Flash update —— Flash update completed. Press any key to reset the system +: Select Screen #: Select Item Enter: Select +/-: Change Opt. F1: General Help F2: Previous Values F3: Optimized Defaults F4: Save All Exit ESC: Exit Version 2.15.1256. Copyright (CI 2017 American Mapsfrends, Inc.

Option #1 as shown below. Then, set Boot Option #1 to [UEFI AP:UEFI: Built-in EFI Shell]. Press to save the settings and exit the BIOS Setup utility.

1. When the UEFI Shell prompt appears, type fs# to change the device directory path. Go to the directory that contains the BIOS package you extracted earlier from Step 6. Enter flash.nsh BIOSname.### at the prompt to start the BIOS update process.

Optio Setup Utility - Copyright (C) 2017 American Megatrends, Inc. Main Advanced Event Loss IPMS Security Boot Save & Exit Boot Configuration Boot mode select [DUAL] LEGACY to EPI Support [Disabled] FIXED BOOT ORDER Priorities Boot Option #1 [DEFI HP/UEFI: No...] Boot Option #2 [COM/DVD] Boot Option #3 [USB Hard Disk] Boot Option #4 [USB CD/DVI] Boot Option #5 [USB Key/SanDisk] Boot Option #6 [USB Flopy] Boot Option #7 [USB Lan] Boot Option #8 [Network: IBM GE SI...] Boot Option #9 [UEFI Hard Disk] Boot Option #10 [UEFI CD/DVD] Boot Option #11 [UEFI USB Hard Disk] Boot Option #12 [UEFI USB CD/DVI] Boot Option #13 [UEFI USB Key/UEFI...] Boot Option #14 [UEFI USB Flopy] Boot Option #15 [UEFI USB Lan] Boot Option #16 [UEFI Network] Boot Option #17 [Hera Disk] ▶ Hold New Boot Option Sets the system boot order ++: Select Screen T4: Select Item Enter: Select +/-: Change Opt, F1: General Help F2: Previous Values F3: Optimized Defaults F4: Save & Exit ESC: Exit Version 2.19.126, Copyright (C) 2017 American Megatrends, Inc.

Note: Do not interrupt this process until the BIOS flashing is complete.

UIFI Interactive Shell v2.1 ESK II UIFI v2.50 (American Regatrends, 0x008900C) Mapping table FS0: FILE0(3):H0D0(1):BLK1; PtROOT(0x0)/PCI(0x14,0x0)/USB(0x11,0x0)/NOI,HDR,0xG7901D2,0x806,0x1 OR3592) BLK0: File(s): PtROOT(0x0)/PCI(0x14,0x0)/USB(0x11,0x0) Press ESR to 1 seconds to skills startup.msh or any other key to continue. Shell: FSO FS0: VAPLDOS FS0:VAPLDOS> CD SAPMHE2_0362012 FS0:VAPLDOS\SAPMHE2_0362017\flash.msh X13P07.SX4

1. The screen above indicates that the BIOS update process is complete. When you see the screen above, unplug the AC power cable from the power supply, clear CMOS, and plug

Done. [ Access Ows: Part Ex ] Reads INDEX 8x51: 0x18 Done. ********************************************************************** * Program BIOS and ME (including RDT) regions... ********************************************************************** AVI Firmware Update Utility v5.49.01.1317 Copyright ©2017 American Registrends Inc. All Rights Reserved. OPUD = 50652 Reading Flash ....... done - ME Data Size checking . ok - PFS checksums ....... ok - Check BookLayout ....... ok Erasing Boot Block ....... done Updating Boot Block ....... done Verifying Boot Block ....... done Froiling Main Block ....... 0x00152000 (00)

the AC power cable in the power supply again to power on the system.

1. Press continuously to enter the BIOS Setup utility.

Verifying HCB Block ....... done - Update success for FDR - Update success for SE. - Successful Update Recovery Loader to (PRx!! - Successful Update MFSB11- - Successful Update RTM11- - Successful Update NFX, [081 and 19021] - Successful Update FLOD and UTOX11 - ML entire image update success!! WAVRNDG : System must power-off to have the changes take effect! Moving FS01:AFUD005/SKJPME2_03162027/40x64,efi -> FS01:AFUD005/SKJPME2_03162017/f df.sac - [cA] Moving FS01:AFUD005/SKJPME2_03162027/aufemfix4,efi -> FS01:AFUD005/SKJPME2_0316201 ?afuefu1.sac - [cA] ****************************************************************************************** * * Please ignore this 'shell cannot read from file - device error' * Warning message due to it does not impact flashing process. * ****************************************************************************************** initating ' ' Delete successful. FSb.-> _

1. Press to load the default settings.
2. After loading the default settings, press to save the settings and exit the BIOS Setup utility.

7.6 CMOS Clear

JBT1 is used to clear CMOS, which will also clear any passwords. Instead of pins, this jumper consists of contact pads to prevent accidentally clearing the contents of CMOS.

To Clear CMOS

1. First power down the system completely.
2. Remove the node from the chassis to access the motherboard.
3. Remove the onboard battery from the motherboard.
4. Short the CMOS pads with a metal object such as a small screwdriver for at least four seconds.
5. Remove the screwdriver or shorting device.
6. Re-install the battery.
7. Replace the cover, reconnect the power cords and power on the system.

Notes: Clearing CMOS will also clear all passwords.

Do not use the PW_ON connector to clear CMOS.

JBT1 contact pads

7.7 Where to Get Replacement Components

If you need replacement parts for your system, to ensure the highest level of professional service and technical support, purchase exclusively from our Supermicro Authorized Distributors/System Integrators/Resellers. A list can be found at: http://www.supermicro.com. Click the "Where to Buy" tab.

7.8 Reporting an Issue

Technical Support Procedures

Before contacting Technical Support, please take the following steps. If your system was purchased through a distributor or reseller, please contact them for troubleshooting services. They have the best knowledge of your specific system configuration.

1. Please review the Troubleshooting Procedures in this manual and Frequently Asked Questions on our website before contacting Technical Support.
2. BIOS upgrades can be downloaded from our website. Note: Not all BIOS can be flashed depending on the modifications to the boot block code.

3. If you still cannot resolve the problem, include the following information when contacting us for technical support:

4. System, motherboard, and chassis model numbers and PCB revision number

5. BIOS release date/version (this can be seen on the initial display when your system first boots up)
6. System configuration

An example of a Technical Support form is posted on our website. Distributors: For immediate assistance, please have your account number ready when contacting our technical support department by email.

Returning Merchandise for Service

A receipt or copy of your invoice marked with the date of purchase is required before any warranty service will be rendered. You can obtain service by calling your vendor for a Returned Merchandise Authorization (RMA) number. When returning to the manufacturer, the RMA number should be prominently displayed on the outside of the shipping carton, and mailed prepaid or hand-carried. Shipping and handling charges will be applied for all orders that must be mailed when service is complete.

For faster service, RMA authorizations may be requested online (http://www.supermicro.com/support/rma/).

Whenever possible, repack the chassis in the original Supermicro carton, using the original packaging material. If these are no longer available, be sure to pack the chassis securely, using packaging material to surround the chassis so that it does not shift within the carton and become damaged during shipping.

This warranty only covers normal consumer use and does not cover damages incurred in shipping or from failure due to the alteration, misuse, abuse or improper maintenance of products.

During the warranty period, contact your distributor first for any product problems.

Vendor Support Filing System

For issues related to Intel, use the Intel IPS filing system:

https://www.intel.com/content/www/us/en/design/support/ips/training/welcome.html

For issues related to Red Hat Enterprise Linux, since it is a subscription based OS, contact your account representative.

7.9 Feedback

Supermicro values your feedback as we strive to improve our customer experience in all facets of our business. Please email us at [e-mail] to provide feedback on our manuals.

Appendix A

Standardized Warning Statements for AC Systems

About Standardized Warning Statements

The following statements are industry standard warnings, provided to warn the user of situations which have the potential for bodily injury. Should you have questions or experience difficulty, contact Supermicro's Technical Support department for assistance. Only certified technicians should attempt to install or configure components.

Read this appendix in its entirety before installing or configuring components in the Supermicro chassis.

These warnings may also be found on our website at http://www.supermicro.com/about/policies/safety_information.cfm.

Warning Definition

Warning! This warning symbol means danger. You are in a situation that could cause bodily injury. Before you work on any equipment, be aware of the hazards involved with electrical circuitry and be familiar with standard practices for preventing accidents.

警告の定義

この警告サインは危険を意味します。

Installation Instructions

Warning! Read the installation instructions before connecting the system to the power source.

設置手順書

Warning! This product relies on the building's installation for short-circuit (overcurrent) protection. Ensure that the protective device is rated not greater than: 250 V, 20 A.

サーキット・ブレーカー

Power Disconnection Warning

Warning! The system must be disconnected from all sources of power and the power cord removed from the power supply module(s) before accessing the chassis interior to install or remove system components.

CAUTION: This unit has redundant power sources. Please disconnect all the power cords before servicing.

電源切断の警告

Equipment Installation

Warning! Only trained and qualified personnel should be allowed to install, replace, or service this equipment.

機器の設置

Warning! This unit is intended for installation in restricted access areas. A restricted access area can be accessed only through the use of a special tool, lock and key, or other means of security. (This warning does not apply to workstations).

アクセス制限区域

Warning! There is the danger of explosion if the battery is replaced incorrectly. Replace the battery only with the same or equivalent type recommended by the manufacturer. Dispose of used batteries according to the manufacturer's instructions

電池の取り扱い

Redundant Power Supplies

Warning! This unit might have more than one power supply connection. All connections must be removed to de-energize the unit.

冗長電源装置

Warning! Hazardous voltage or energy is present on the backplane when the system is operating. Use caution when servicing.

バックプレーンの電圧

Comply with Local and National Electrical Codes

Warning! Installation of the equipment must comply with local and national electrical codes.

地方および国の電気規格に準拠

Warning! Ultimate disposal of this product should be handled according to all national laws and regulations.

製品の廃棄

Warning! Hazardous moving parts. Keep away from moving fan blades. The fans might still be turning when you remove the fan assembly from the chassis. Keep fingers, screwdrivers, and other objects away from the openings in the fan assembly's housing.

ファン・ホットスワップの警告

Power Cable and AC Adapter

Warning! When installing the product, use the provided or designated connection cables, power cables and AC adaptors. Using any other cables and adaptors could cause a malfunction or a fire. Electrical Appliance and Material Safety Law prohibits the use of UL or CSA-certified cables (that have UL/CSA shown on the cord) for any other electrical devices than products designated by Supermicro only.

電源コードとACアダプター

System Specifications

Processors (per node)

Supports dual 4th Gen Intel Xeon Scalable Processors (in Socket E LGA4677) with four UPIs (max 16GT/s) and a TDP (thermal design power) of up to 350 W per node. Supports SP XCC, SP MCC, and HBM SKU on the X13DET-B. Note: Refer to the motherboard specifications pages on our website for updates to supported processors.

Chipset (per node)

Intel C741 PCH

BIOS (per node)

AMI BIOS

ACPI 3.0 or later, PCI firmware 4.0 support, BIOS rescue hot-key, SPI dual/quad speed support, riser card auto detection support, RTC (Real Time Clock) wakeup, and SMBIOS 3.0 or later

Memory (per node)

Supports up to 4 TB RDIMM DDR5 (288-pin) ECC memory with speeds up to 4800 MT/s in 16 DIMM slots

Storage Drives

Front hot-swappable drives include:

24 NVMe drives for SYS-221BT-HNR

24 NVMe/SATA drives for SYS-221BT-HNTR

24 NVMe/SAS drives for SYS-221BT-HNC8R

24 NVMe/SAS drives for SYS-221BT-HNC9R

Internal M.2 drives per node include:

Carrier with two M.2 NVMe Gen 4 SSDs in the 22110 form factor

Carrier with two M.2 NVMe Gen 3 SSDs in the 2280 form factor

Internal Connector:

VROC key header

PCI Expansion Slots (per node)

Two PCIe 5.0 x16 slots

Networking (per node)

One AIOM or any compliant OCP 3.0 SFF Network Interface Card

One dedicated LAN port for BMC

Input/Output (per node)

Two USB 3.1 ports

One VGA port

Motherboard (per node)

X13DET-B (WxL) 7.4" x 18.86" (187.96 mm x 479.04 mm)

Chassis

CSE-217BQ2-R3K04P; 2U rackmount; (WxHxD) 17.6 x 3.5 x 28.8 in. (447 x 88 x 730 mm)

System Cooling

Four 8-cm mid-chassis fans, two CPU heat sinks per node, and one air shroud per node

Power Supply

Model: PWS-3K04A-1R, 3000 W redundant module, 80Plus Titanium level

Input voltages, input currents

1400W: 100-127Vac, 16-13A

2880W: 200-207Vac, 16-15.5A

3000W: 208-240Vac, 16-14A

3000W: 240Vdc, 14A (CQC only)

Rated Frequency

50-60Hz

Output DC Voltage: Input current

+12 V: 116A (1400W)

+12 V: 240A (2880W)

+12 V: 250A (3000W)

Output DC Voltage: Input current

+12 V standby: 4.5A (1400W)

+12 V standby: 4.5A (2880W)

+12 V standby: 4.5A (3000W)

Operating Environment

Operating Temperature: 10° to 35° C (50° to 95° F)

Non-operating Temperature: -40^ to 60^ C ( -40^ to 140^ F)

Operating Relative Humidity: 8% to 90% (non-condensing)

Non-operating Relative Humidity: 5% to 95% (non-condensing)

Regulatory Compliance

FCC, ICES, CE, VCCI, RCM, UKCA, NRTL, CB

Applied Directives, Standards

EMC/EMI: 2014/30/EU (EMC Directive)

Electromagnetic Compatibility Regulations 2016

FCC Part 15 Subpart B

ICES-003

VCCI-CISPR 32

AS/NZS CISPR 32

BS/EN55032

BS/EN55035

BS/EN 61000-3-2

BS/EN 61000-3-3

BS/EN 61000-4-2

BS/EN 61000-4-3

BS/EN 61000-4-4

BS/EN 61000-4-5

BS/EN 61000-4-6

BS/EN 61000-4-8

BS/EN 61000-4-11

Green Environment:

2011/65/EU (RoHS Directive)

EC 1907/2006 (REACH)

2012/19/EU (WEEE Directive)

Product Safety: 2014/35/EU (LVD Directive)

Electrical Equipment (Safety) Regulations 2016

UL/CSA 62368-1 (USA and Canada)

BS/IEC/EN 62368-1

Perchlorate Warning

California Best Management Practices Regulations for Perchlorate Materials: This Perchlorate warning applies only to products containing CR (Manganese Dioxide) Lithium coin cells. "Perchlorate Material-special handling may apply. See www.dtsc.ca.gov/hazardouswaste/perchlorate"

Declaration of the Presence Condition of the Restricted Substances Marking

設備名稱: 伺服器 / Server 型號 (型式): 217B-R30X13Equipment name Type designation (Type)(系列型號: 217B-30, 217B-R22X13, 217B - 22, 827B-R30X13, 827B-30, 827B -R22X13, 827B -22,SYS-221BT-HNTR, SYS-221BT-HNR, SYS-221BT-HNC8R, SYS-221BT-HNC9R, SYS-621BT-HNTR,SYS-621BT-HNC8R)
單元Unit	限用物質及其化學符號Restricted substances and its chemical symbols
	鉛Lead(Pb)	汞Mercury(Hg)	鎘Cadmium(Cd)	六價鉻Hexavalent chromium( Cr^-6 )	多溴聯苯Polybrominated biphenyls(PBB)	多溴二苯醚Polybrominated diphenyl ethers(PBDE)
機殼(Chassis)	○	○	○	○	○	○
機殼風扇(Chassis Fan)	-	○	○	○	○	○
線材(Cable)	○	○	○	○	○	○
主機板(Motherboard)	-	○	○	○	○	○
電源供應器(Power Supply)	-	○	○	○	○	○
硬碟(SSD)	-	○	○	○	○	○
電源背板(PDB)	-	○	○	○	○	○
附加卡(Add-on card)	-	○	○	○	○	○
備考1. “超出0.1 wt %”及“超出0.01 wt %”係指限用物質之百分比含量超出百分比含量基準值。Note 1: “Exceeding 0.1 wt %” and “exceeding 0.01 wt %” indicate that the percentage content of the restricted substance exceeds the reference percentage value of presence condition.備考2. *o*係指該項限用物質之百分比含量未超出百分比含量基準值。Note 2: “○” indicates that the percentage content of the restricted substance does not exceed the percentage of reference value of presence.備考3. * - ”係指該項限用物質為排除項目。Note 3: The “-” indicates that the restricted substance corresponds to the exemption.

BUREAU OF STANDARDS, METROLOGY AND INSPECTION, MINISTRY OF ECONOMIC AFFAIRS

商品驗證登錄證書

CERTIFICATE OF THE REGISTRATION OF PRODUCT CERTIFICATION

for Registration of Product

The application made by

SUPER MICRO COMPUTER, INC. TAIWAN

Certification has been reviewed and found to be in compliance with related regulations. Therefore, registration is granted with the

Product Safety Mark and the Identification No.

R36199

. Details of the registration are follows

申請人：美超微電腦股份有限公司

统一編號：12729477

Applicant

Uniform No.

地址：新北市中和區建一路150號3樓

Address

生產廠場：詳如附表

Factory

廠址：詳如附表

Factory Address

商品種類名稱：

Type/name of product

This certificate is issued by the BSMI. (No. 4, Sec. 1, Jinan Rd., Zhongzheng Dist., Taipei City 100, Taiwan)

BUREAU OF STANDARDS, METROLOGY AND INSPECTION, MINISTRY OF ECONOMIC AFFAIRS

商品驗證登錄證書

CERTIFICATE OF THE REGISTRATION OF PRODUCT CERTIFICATION

BUREAU OF STANDARDS, METROLOGY AND INSPECTION,

MINISTRY OF ECONOMIC AFFAIRS

商品驗證登錄證書

CERTIFICATE OF THE REGISTRATION OF PRODUCT CERTIFICATION