SuperServer SYS-620BT-DNTR - Server Supermicro - Free user manual and instructions

USER MANUAL SuperServer SYS-620BT-DNTR Supermicro

Front view of a rack-mounted server with multiple drive bays and orange ports (no visible text or labels)

USER'S MANUAL

Revision 1.0

The information in this User's Manual has been carefully reviewed and is believed to be accurate. The vendor assumes no responsibility for any inaccuracies that may be contained in this document, and makes no commitment to update or to keep current the information in this manual, or to notify any person or organization of the updates. Please Note: For the most up-to-date version of this manual, please see our website at www.supermicro.com.

Super Micro Computer, Inc. ("Supermicro") reserves the right to make changes to the product described in this manual at any time and without notice. This product, including software and documentation, is the property of Supermicro and/or its licensors, and is supplied only under a license. Any use or reproduction of this product is not allowed, except as expressly permitted by the terms of said license.

IN NO EVENT WILL Super Micro Computer, Inc. BE LIABLE FOR DIRECT, INDIRECT, SPECIAL, INCIDENTAL, SPECULATIVE OR CONSEQUENTIAL DAMAGES ARISING FROM THE USE OR INABILITY TO USE THIS PRODUCT OR DOCUMENTATION, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. IN PARTICULAR, SUPER MICRO COMPUTER, INC. SHALL NOT HAVE LIABILITY FOR ANY HARDWARE, SOFTWARE, OR DATA STORED OR USED WITH THE PRODUCT, INCLUDING THE COSTS OF REPAIRING, REPLACING, INTEGRATING, INSTALLING OR RECOVERING SUCH HARDWARE, SOFTWARE, OR DATA.

Any disputes arising between manufacturer and customer shall be governed by the laws of Santa Clara County in the State of California, USA. The State of California, County of Santa Clara shall be the exclusive venue for the resolution of any such disputes. Supermicro's total liability for all claims will not exceed the price paid for the hardware product.

FCC Statement: This equipment has been tested and found to comply with the limits for a Class A or Class B digital device pursuant to Part 15 of the FCC Rules. These limits are designed to provide reasonable protection against harmful interference when the equipment is operated in industrial environment for Class A device or in residential environment for Class B device. This equipment generates, uses, and can radiate radio frequency energy and, if not installed and used in accordance with the manufacturer's instruction manual, may cause harmful interference with radio communications. Operation of this equipment in a residential area is likely to cause harmful interference, in which case you will be required to correct the interference at your own expense.

California Best Management Practices Regulations for Perchlorate Materials: This Perchlorate warning applies only to products containing CR (Manganese Dioxide) Lithium coin cells. "Perchlorate Material-special handling may apply. See www.dtsc.ca.gov/hazardouswaste/perchlorate".

WARNING: This product can expose you to chemicals including lead, known to the State of California to cause cancer and birth defects or other reproductive harm. For more information, go to www.P65Warnings.ca.gov.

The products sold by Supermicro are not intended for and will not be used in life support systems, medical equipment, nuclear facilities or systems, aircraft, aircraft devices, aircraft/emergency communication devices or other critical systems whose failure to perform be reasonably expected to result in significant injury or loss of life or catastrophic property damage. Accordingly, Supermicro disclaims any and all liability, and should buyer use or sell such products for use in such ultra-hazardous applications, it does so entirely at its own risk. Furthermore, buyer agrees to fully indemnify, defend and hold Supermicro harmless for and against any and all claims, demands, actions, litigation, and proceedings of any kind arising out of or related to such ultra-hazardous use or sale.

Manual Revision 1.0

Release Date: April 21, 2021

Unless you request and receive written permission from Super Micro Computer, Inc., you may not copy any part of this document. Information in this document is subject to change without notice. Other products and companies referred to herein are trademarks or registered trademarks of their respective companies or mark holders.

Copyright © 2021 by Super Micro Computer, Inc.

All rights reserved.

Printed in the United States of America

Preface

About this Manual

This manual is written for professional system integrators and PC technicians. It provides information for the installation and use of the server. Installation and maintenance should be performed by experienced technicians only.

Please refer to the 620BT-D Series server specifications page on our website for updates on supported memory, processors and operating systems (http://www.supermicro.com).

Notes

For your system to work properly, please follow the links below to download all necessary drivers/utilities and the user's manual for your server.

• Supermicro product manuals: http://www.supermicro.com/support/manuals/
• Product drivers and utilities: https://www.supermicro.com/wdl
• Product safety info: http://www.supermicro.com/about/policies/safety_information.cfm

If you have any questions, please contact our support team at:

[e-mail]

This manual may be periodically updated without notice. Please check the Supermicro website for possible updates to the manual revision level.

Secure Data Deletion

A secure data deletion tool designed to fully erase all data from storage devices can be found on our website: https://www.supermicro.com/about/policies/disclaimer.cfm?url=/wdl/utility/Lot9_Secure_Data_Deletion_Utility/

Warnings

Special attention should be given to the following symbols used in this manual.

Warning! Indicates important information given to prevent equipment/property damage or personal injury.

Warning! Indicates high voltage may be encountered when performing a procedure.

Contents

Chapter 1 Introduction

1.1 Overview....9
1.2 System Features ....10

Control Panel....11

Rear View....12

AIOM Network Ports ....13

Power Supply Options....13

Node Tray....14

1.3 System Architecture ....16

Main Components....16

System Block Diagram....17

1.4 Motherboard Layout....19

Quick Reference Table....20

Motherboard Block Diagram ....21

Chapter 2 Server Installation

2.1 Overview....22
2.2 Unpacking the System....22
2.3 Preparing for Setup....22

Choosing a Setup Location....22

Rack Precautions....23

Server Precautions....23

Rack Mounting Considerations....23

Ambient Operating Temperature....23
Airflow 24
Mechanical Loading....24
Circuit Overloading....24
Reliable Ground....24

2.3 Rack Mounting Instructions....26

Overview of the Rack Rails....26

Adjusting the Rail Length ....26

Installing the Rails on a Rack....27

Chassis Installation....28

Chapter 3 Maintenance and Component Installation

3.1 Removing Power ....29
3.2 Accessing the System....30

Removing a Computing Node Drawer....30

Removing the Chassis Cover ....31

3.3 Processor and Heatsink 32

Prepare the System 32

ESD Precautions....32

Installation Overview....33

Removal Overview....33

Heatsink Overview 34

Create the Processor Carrier Assembly ....35

Create the Processor Heatsink Module (PHM) 37

Prepare the Socket ....38

Install the PHM....39

Remove the PHM from the Motherboard....41

Remove the Carrier Assembly from the Heatsink 42

Remove the Processor from the Carrier Assembly 43

3.4 Memory....44

Memory Support for the 3rd Gen Intel Xeon Scalable Processors....44

Memory Population Table for the 3rd Gen Intel Scalable Processor....45

Intel Optane PMem 200 Series Memory Population Table....46

DIMM Installation 48

DIMM Removal 48

3.5 Motherboard Battery....49

3.6 Storage Drives....50

Drive Carriers....50

Drive Configuration ....51

Installing Drives....52

Hot-Swap for NVMe Drives....54

Checking the Temperature of an NVMe Drive ....55

Adding an Expansion Card....56

AIOM Card 59

3.7 System Cooling....60

Fans 60

Installing Air Shroud....62

Checking the Airflow....62

3.8 Power Supply 63
3.9 Cable Routing Diagram....64
3.10 BMC Reset....65

Chapter 4 Motherboard Connections

4.1 Power Connections ....66
4.2 Headers and Connectors....67
4.3 Input/Output Ports 69
4.4 Jumpers....72

How Jumpers Work....72

4.5 LED Indicators....74

Chapter 5 Software

5.1 Microsoft Windows OS Installation....75
5.2 Driver Installation....77
5.3 SuperDoctor® 5....78
5.4 BMC....79

BMC ADMIN User Password 79

Chapter 6 Optional Components

6.1 Optional Parts List....80
6.2 M.2 NVMe Carrier Card ....80
6.3 GPU Support....80

Chapter 7 Troubleshooting and Support

7.1 Information Resources ....81
Website 81
Direct Links for the 620BT-D Series System....81
Direct Links for General Support and Information 81
7.2 Baseboard Management Controller (BMC)....82
7.3 Troubleshooting Procedures 83

No Power 83

System Boot Failure 84

Memory Errors 84

Losing the System's Setup Configuration....84

When the System Becomes Unstable 85

7.4 Crash Dump Using BMC....87

7.5 UEFI BIOS Recovery ....88

Overview 88

Recovering the UEFI BIOS Image....88

Recovering the Main BIOS Block with a USB Device....88

7.6 CMOS Clear....93

7.7 Where to Get Replacement Components....94

7.8 Reporting an Issue....94

Technical Support Procedures....94

Returning Merchandise for Service....94

Vendor Support Filing System 95

7.9 Feedback....95

Appendix A Standardized Warning Statements for AC Systems

Appendix B System Specifications

Appendix C CPU-Based RAID for NVMe

Contacting Supermicro

Headquarters

Address: Super Micro Computer, Inc.

980 Rock Ave.

San Jose, CA 95131 U.S.A.

Tel: [phone]

Fax: [phone]

Email: [e-mail] (General Information)

[e-mail] (Technical Support)

Website: www.supermicro.com

Europe

Address: Super Micro Computer B.V.

's-Hertogenbosch, The Netherlands

Tel: [phone]

Fax: [phone]

Email: [e-mail] (General Information)

[e-mail] (Technical Support)

[e-mail] (Customer Support)

Website: www.supermicro.nl

Asia-Pacific

Address: Super Micro Computer, Inc.

3F, No. 150, Jian 1st Rd.

Zhonghe Dist., New Taipei City 235

Taiwan (R.O.C)

Tel: [phone]

Fax: [phone]

Email: [e-mail]

Website: www.supermicro.com.tw

Chapter 1

Introduction

1.1 Overview

This chapter provides a brief outline of the functions and features of the SuperServer 620BT-D Series. It is based on the X12DPT-B6 motherboard and the CSE-827BD-R2K22P chassis.

The following provides an overview of the specifications and capabilities.

System Overview
Motherboard	X12DPT-B6
Chassis	CSE-827BD-R2K22P
Processor Support	Supports dual Intel Xeon Scalable Family 3rd Gen/4th Gen Series Processors (in Socket P+) with a thermal design power (TDP) of up to 250W and 3 UPIs (UltraPath Interconnect) of up to 11.2 GT/s per node.
Chipset	Intel PCH C621A (LBG-R)
Memory	16 DRAM DIMMs plus two PMems (Intel® OptaneTM PMem 200 Series) per node. Up to 4TB RDIMM, DDR4-3200MHz in 16 memory slots
Drive Support	Front hot-swappable drives include:12 NVMe/SATA drives for 620BT-DNTR12 NVMe/SAS/SATA drives for 620BT-DNC8RInternal M.2 drives per node include:Default carrier with two M.2 NVMe/SATA SSDsOptional carrier with two M.2 NVMe SSDs and Marvell HW RAIDInternal Connector:VROC key header
Expansion Slots	Two PCIe 4.0 x8 slots and one PCIe 4.0 x16 for each node
Networking	One AIOM or any compliant OCP 3.0 SFF Network Interface Card per nodeOne dedicated LAN port for BMC per node
I/O Ports	Two USB 3.0 ports per nodeOne VGA port per node
System Cooling	Four 8-cm mid-chassis fansTwo CPU heatsinks per nodeOne air shroud per node
Power	Redundant 2200W modules, 80Plus level Titanium
Form Factor	2U rackmount; (WxHxD) 17.6 x 3.5 x 30.5 in. (446 x 88 x 774 mm)

Note1: A Quick Reference Guide can be found on the product page of the Supermicro website.
Note2: The following safety models associated with the 620BT-D Series have been certified as compliant with CSA or UL models: 827B-22 and 827B-R22X12.

1.2 System Features

The CSE-827BD-R2K22P is a 2U chassis that supports 12 front hot-swappable drives and two rear hot-pluggable nodes.

Front View

The chassis front offers access to the storage drives, a control panel for each node, a pull-out service tag and two thumbscrews.

Control Panel for Node A Control Panel for Node B A2 A1 A5 B2 B5 A0 A4 B1 B4 A3 B0 B3 Node A Drive Bays Node B Drive Bays Service Tag Thumbscrew Thumbscrew

Figure 1-1. Front View

System Features: Front
Feature Description
Control Panels	Four control panels with labels for each node. The four control panels are located as follows: node A bottom left, node B top left, node C bottom right, and node D top right.
Service Tag Pull-out service tag with BMC password label
Drive Bays 12 hot-swappable drive bays (six per node)
Thumbscrews Two thumbscrews to secure the server onto the rack

Control Panel

Node Label NIC LED Information LED Power Button i BMC Button/UID LED UID

Figure 1-2. Control Panel

Control Panel Features
Feature Description
Node Label Label	with the name of the node that is connected to the control panel.
Power Button	The main power switch applies or removes primary power from the power supply to the server but maintains standby power.
NIC LED Indicates	network activity on the LAN when flashing.
Information LED U	Universal information LED (see table below for details).
BMC Button/UID LED	The BMC reset button resets the BMC firmware when pressed. The unit identification (UID) button turns on or off the blue light function of the Information LED and a blue LED on the rear of the chassis. These are used to locate the server in large racks and server banks.

Information LED
Status Description
Continuously on and red An	overheat condition has occurred.
Blinking red (1Hz) Fan failure	, check for an inoperative fan.
Blinking red (0.25Hz) Power	failure, check for a non-operational power supply.
Solid blue	Local UID has been activated. Use this function to locate the server in a rackmount environment.
Blinking blue Remote UID is	on. Use this function to identify the server from a remote location.

Rear View

Node Handle Node Handle Power Supply B1 B2 B1 A A2 A1 A Node B Dual Function Switch BMC LAN USB Ports VGA Port Node A

Figure 1-3. System: Rear View

System Features: Rear
Feature Description
Nodes A and B Independent computing nodes
Power Supplies Two 2200W redundant power supplies
VGA Port Video port
USB Ports Two USB 3.0 ports
LAN Port Dedicated BMC LAN port
Dual Function Switch	A switch that can function as either a UID LED switch or a BMC reset switch.See Chapter 4 for a description of the dual function switch.
Node Handles Two handles per node supporting node tray removal

Expansion Slot Locations
Item Description
1 PCIe 4.0 x8 Low	Profile slot1 (CPU1) in each node
2 PCIe 4.0 x8 Low	Profile slot2 (CPU1) in each node
1 PCIe 4.0 x16 slot (CPU2) in each node
A AIOM PCIe 4.0 x16 slot (CPU1) in each node

CPU1 CPU2

AIOM Network Ports

Network ports are provided by the AIOM card, which offers several choices of connection speeds and types.

AIOM Networking Add-on Card Options
Speed Ports Add-on Card Part Number
GbE	Two RJ45 AOC-AG-i2M
	Four RJ45 AOC-AG-i4M
	Four SFP AOC-AG-i4SM
10 G SFP+	Two SFP+ AOC-ATG-i2SM
	Four SFP+ AOC-ATG-i4SM
	Two SFP+ & two RJ45 AOC-ATG-i2T2SM
10GBase-T Two RJ45 AOC-ATG-i2TM
25 GbE	Two SFP28 & two RJ45	AOC-AH25G-m2S2TM
	One SFP28	MCX623432AC-ADAB
	Two SFP28	MCX4621A-ACAB
100 GbE	QSFP56	MCX623435AC-VDAB

Updates: https://www.supermicro.com/en/support/resources/aoc/aiom

Power Supply Options

Power Supply Module Options
Watts Part Number	80Plus Level
2200 PWS-2K22A-1R	Titanium

Node Tray

The chassis contains two separate computing node drawers, each with its own motherboard.

Storage Adapter Card M.2 Card with Riser Dual Intel® Xeon® Scalable Processors 16 DDR4 DIMM slots plus two PMEM slots SATA Cable AIOM Slot

Figure 1-4. 620BT-DNTR Node Tray

Storage Adapter M.2 Card with Riser Dual Intel® Xeon® Scalable Processors 16 DDR4 DIMM slots plus two PMEM slots SAS Controller Card for 620BT-DNC8R AIOM Slot

Figure 1-5. 620BT-DNC8R Node Tray

1.3 System Architecture

This section covers the printed circuit board (PCB) locations and system block diagrams.

Main Components

Storage Backplane M.2 Card with Riser AIOM PCIe x8 Riser Card Single Node Storage Adapter SAS Controller Card for 620BT-DNC8R Motherboard

Figure 1-6. Main Component Locations

System Block Diagram

The block diagram below shows the connections and relationships between the subsystems and major components of the overall system.

System architecture diagram showing data flow between CPU, memory, and peripheral components like BMC AST2600, SSSB, and I-SATAO-7.

Figure 1-7. 620BT-DNTR System Block Diagram

System architecture diagram showing data flow between BMC AST2600 chip, CPU cores, I/O interfaces, and peripheral components like BIOS, DDR4, and SXTAOS.

Figure 1-8. 620BT-DNC8R System Block Diagram

1.4 Motherboard Layout

Below is a layout of the X12DPT-B6 motherboard with jumper, connector and LED locations shown. See the table on the following page for descriptions. For detailed descriptions, pinout information and jumper settings, refer to Chapter 4 or the Motherboard Manual.

UID/BMC RESET UID_LED1 JPFR3 JPME1 JPFR2 LED2 BMC_LAN1 JPGF1 S-SATA2-5 Slot1 BT1 BJA1 VGA1 USB0/1(3.0) JUSB1 JPGF1 JPGF1 JPGF1 JPGF1 JPGF1 JPGF1 JPGF1 JPGF1 JPGF1 JPGF1 JPGF1 JPGF1 JPGF1 JPGF1 JPGF1 JPGF1 JPGF1 JPGF1 JPGF1 JPGF1 JPGM1 JPTM1 JCPLD1 JAOM1 JAKIOM AID CPU1 PCIE 4.0 X16 SATA1 SLOT2A SATA1_SATA 4-7 PCH JBT1 CMOS CLEAR JBT1 FAN3 P1-DIMMC2 P1-DIMMC1 P1-DIMMD1 P1-DIMMA1 P1-DIMMB1 SXB4 SXB3 P2-DIMMG2 P2-DIMMG1 P2-DIMMH1 P2-DIMME1 P2-DIMMF1 SXB2 JPWR4 JPWR3 SXB1 JPWR2 JPWR1 FC CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 CPU9 CPU10 CPU11 CPU12 CPU13 CPU14 CPU15 CPU16 CPU17 CPU18 CPU19 CPU20 CPU21 CPU22 CPU23 CPU24 CPU25 CPU26 CPU27 CPU28 CPU29 CPU30 CPU31 CPU32 CPU33 CPU34 CPU35 CPU36 CPU37 CPU38 CPU39 CPU40 CPU41 CPU42 CPU43 CPU44 CPU45 CPU46 CPU47 CPU48 CPU49 CPU50 CPU51 CPU52 CPU53 CPU54 CPU55 CPU56 CPU57 CPU58 CPU59 CPU60 CPU61 CPU62 CPU63 CPU64 CPU65 CPU66 CPU67 CPU68 CPU69 CPU70 CPU71 CPU72 CPU73 CPU74 CPU75 CPU76 CPU77 CPU78 CPU79 CPU80 CPU81 CPU82 CPU83 CPU84 CPU85 CPU86 CPU87 CPU88 CPU89 CPU90 CPU91 CPU92 CPU93 CPU94 CPU95 CPU96 CPU97 CPU98 CPU99 CPU100

Figure 1-9. Motherboard Layout

Quick Reference Table

Jumper Description Default Setting

JBT1 CMOS Clear Open (Normal)

JPME1 ME Manufacturing Recovery Pins 1-2 (Normal)

LED Description Status

UID_LED1 Unit Identifier (UID) LED Solid Blue: Unit Identified
LED2 BMC Heartbeat LED	Blinking Green: BMC NormaSolid Greene: (during BMC Reset or a Cold Reboot)I

Connector Description

BT1 Onboard battery
COM1 I/O COM port
FAN3, FAN4	CPU/System fan headers (FAN1 & FAN2 in the HDD backplane)
BMC_LAN1	Dedicated BMC LAN port
I-SATA 0~7 (SATA1)	Intel® PCH SATA 3.0 ports (with RAID 0, 1, 5, 10)
S-SATA 0~1, S-SATA 2~5	S-SATA 3.0 connectors supported by the Intel PCH
JNCSI NC-SI (Network Controller Sideband Interface) connector (See Note 1 below)
JPWR1/JPWR2/JPWR3/JPWR4	6-pin power connectors
JRK1 (RAID_KEY)	Intel VROC key header for NVMe RAID (See Note 2 below)
JAIOM	AIOM (CPU1 PCIe 4.0x16) networking slot
JTPM1	Trusted Platform Module/Port 80 connector
JVGA1	VGA port
SLOT1	PCIe 4.0 x16 slot (right hand riser) supported by CPU1
SLOT2	PCIe 4.0 x16 slot (left hand riser, via cable) supported by CPU2
SXB1	PCIe 4.0 x16 slot supported by CPU2
SXB2	PCIe 4.0 x8 slot supported by CPU2
SXB3	PCIe 4.0 x16 slot supported by CPU1
SXB4	PCIe 4.0 x8 slot supported by CPU1 for SMCI-proprietary storage devices
SXB5 (S~SATA0~1) PCIe 4.0 x8 slot supported by CPU1 for dual hybrid M.2
UID/BMC RESET (JUIDB1)	Unit Identifier (UID) & BMC Reset switch
USB0/1 (3.0)	Two USB 3.0 connectors

Note 1: For detailed instructions on how to configure Network Interface Card (NIC) settings, please refer to the Network Interface Card Configuration User's Guide posted on the web page under the link: http://www.supermicro.com/support/manuals/.

Note 2: For detailed instructions on how to configure VROC RAID settings, please refer to the VROC RAID Configuration User's Guide posted on the web page under the link: http://www.supermicro.com/support/manuals/.

Motherboard Block Diagram

graph TD
    subgraph CPU
        A["CPU#1"] -->|P0| B["VCCP0 12v"]
        C["SNB CORE"] -->|P1| D["VCCP1 12v"]
        E["DDR4"] -->|P2| F["DDR4 12v"]
        G["PCIE:64"] -->|P3| H["PCIE X16"]
        I["SOCKET ID:0"] -->|P0| J["SOCKET ID:1"]
        K["#0 #1 #2 #3 DMI3"] --> L["PCI-E X16"]
        M["#0 #1 #2 #3 DMI3"] --> N["PCI-E X16"]
        O["SXB4"] --> P["PCI-E X16"]
        Q["SXB5"] --> R["PCI-E X16"]
        S["XSB3"] --> T["PCI-E X16"]
        U["PCI-E X16"] --> V["PCI-E X8"]
        W["PCI-E X16"] --> X["PCI-E X16"]
    end

    subgraph Memory
        Y["PCIH LBG-1G 15W"] --> Z["PCH PCIE #4"]
        AA["NCSI"] --> AB["I-SATAO-7"]
        AC["S-SATA2-5"] --> AD["SATA1"]
        AE["S-SATAO-1"] --> AF["SATA2"]
        AG["USB 2.0"] --> AH["USB 3.0"]
        AI["USB 2.0"] --> AJ["USB"]
    end

    subgraph I-SATAO
        AK["PCH PCIE #5"] --> AL["SPI Header"]
        AM["ESPI"] --> AN["SPI Header"]
    end

    subgraph Remote
        AO["BMC Flash"] --> AP["BIOS"]
    end

    subgraph Control
        AQ["VGA CONN"] --> AR["ROMII/NCSI"]
        AS["COM1 Connector"] --> AT["RGRMII"]
        AU["Temp Sensor NCT7718W"] --> AV["SPI Header"]
        AW["TPM HEADER Debug Card"] --> AX["SPI Header"]
    end

    CPU -->|UPI 10.4/11.2G| I-SATAO
    CPU -->|UPI 10.4/11.2G| AJ-SATAO
    CPU -->|UPI 10.4/11.2G| AK-SATAO
    CPU -->|UPI 10.4/11.2G| AL-SATAO
    CPU -->|UPI 10.4/11.2G| AM-SATAO
    CPU -->|UPI 10.4/11.2G| AN-SATAO
    CPU -->|UPI 10.4/11.2G| AO-BIO
    CPU -->|UPI 10.4/11.2G| AP-BIO
    CPU -->|UPI 10.4/11.2G| AQ-BIO
    CPU -->|UPI 10.4/11.2G| AP-BIO
    CPU -->|UPI 10.4/11.2G| AQ-BIO
    CPU -->|UPI 10.4/11.2G| AP-BIO
    CPU -->|UPI 10.4/11.2G| AQ-BIO
    CPU -->|UPI 15A 6A 6B 6C 6D 6E 6F 6G 6H 6I 6J 6K 6L 6M 6N 7A 7B 7C 7D 7E 7F 7G 7H 7I 7J 7K 7L 7M 7N 7O 7P 7Q 7R 7S 7T 7U 7V 7W 7X 7Y 7Z
    CPU -->|UPI 10.4/11.2G| I-SATAO
    CPU -->|UPI 10.4/11.2G| AJ-SATAO
    CPU -->|UPI 10.4/11.2G| AK-SATAO
    CPU -->|UPI 10.4/11.2G| AL-SATAO
    CPU -->|UPI 9.8/9.8/9.8/9.8/9.8/9.8/9.8/9.8/9.8/9.8/9.8/9.8/9.8/9.8/9.8/9.8/9.8/9.8/9.8/9.8/9.8/9.8/9.8/9.8/9.8/9.8
    end

    subgraph Remote
        AO_BIO
        AO_BIO
        AO_BIO
        AO_BIO
        AO_BIO
        AO_BIO
        AO_BIO
        AO_BIO
        AO_BIO
        AO_BIO
        AO_BIO
        AO_BIO
        AO_BIO
        AO_BIO
        AO_BIO
        AO_BIO
        AO_BIO
        AO_BIO
        AO_BIO
    end

    CPU -->|PCI-E X16| AI["AIOM"]
    CPU -->|PCI-E X16| AJ["AIOM"]

    CPU -->|PCI-E X16| AK_3["PCI-E X16"]

    CPU -->|PCI-E X16| AJ_3["PCI-E X16"]

    CPU -->|PCI-E X16| AK_3_3["PCI-E X16"]

    CPU -->|PCI-E X16| AJ_3_3["PCI-E X16"]

    CPU -->|PCI-E X16| AK_3_3_3["PCI-E X16"]

    CPU -->|PCI-E X16| AJ_3_3_3_3["PCI-E X16"]

    CPU -->|PCI-E X16| AK_3_3_3_3["PCI-E X16"]

    CPU -->|PCI-E X16| AJ_3_3_3_3_3["PCI-E X16"]

    CPU -->|PCI-E X16| AK_3_3_3_3_3["PCI-E X16"]

    CPU -->|PCI-E X8| AL["DIOM"]
    CPU -->|PCI-E X8| AJ["DIOM"]

    CPU -->|PCI-E X8| AK_3_3_3_3_3_3_3_3_3_3_3_3_3_3_3_3_3_3_3_3_3_3_3_3_3_3_3_3_3_3_3_3_3_3_3_3_3_3_3_3_3_3_3_3_3_3_3_3_3_3_5

Figure 1-10. Motherboard Block Diagram

Chapter 2

Server Installation

2.1 Overview

This chapter provides advice and instructions for mounting your system in a server rack. If your system is not already fully integrated with processors, system memory etc., refer to Chapter 3 for details on installing those specific components.

Caution: Electrostatic Discharge (ESD) can damage electronic components. To prevent such damage to PCBs (printed circuit boards), it is important to use a grounded wrist strap, handle all PCBs by their edges and keep them in anti-static bags when not in use.

2.2 Unpacking the System

Inspect the box in which the SuperServer 620BT-D Series was shipped, and note if it was damaged in any way. If any equipment appears damaged, file a damage claim with the carrier who delivered it.

Decide on a suitable location for the rack unit that will hold the server. It should be situated in a clean, dust-free area that is well ventilated. Avoid areas where heat, electrical noise and electromagnetic fields are generated. It will also require a grounded AC power outlet nearby. Be sure to read the precautions and considerations noted in Appendix A.

2.3 Preparing for Setup

The box in which the system was shipped should include the rackmount hardware needed to install it into the rack. Please read this section in its entirety before you begin the installation.

Choosing a Setup Location

• The system should be situated in a clean, dust-free area that is well ventilated. Avoid areas where heat, electrical noise and electromagnetic fields are generated.
• Leave enough clearance in front of the rack so that you can open the front door completely (\~25 inches) and approximately 30 inches of clearance in the back of the rack to allow sufficient space for airflow and access when servicing.
• This product should be installed only in a Restricted Access Location (dedicated equipment rooms, service closets, etc.).

- This product is not suitable for use with visual display workplace devices according to §2 of the German Ordinance for Work with Visual Display Units.

Rack Precautions

• Ensure that the leveling jacks on the bottom of the rack are extended to the floor so that the full weight of the rack rests on them.
• In single rack installations, stabilizers should be attached to the rack. In multiple rack installations, the racks should be coupled together.
• Always make sure the rack is stable before extending a server or other component from the rack.
• You should extend only one server or component at a time - extending two or more simultaneously may cause the rack to become unstable.

Server Precautions

• Review the electrical and general safety precautions in Appendix A.
• Determine the placement of each component in the rack before you install the rails.
• Install the heaviest server components at the bottom of the rack first and then work your way up.
• Use a regulating uninterruptible power supply (UPS) to protect the server from power surges and voltage spikes and to keep your system operating in case of a power failure.
• Allow any drives and power supply modules to cool before touching them.
• When not servicing, always keep the front door of the rack and all covers/panels on the servers closed to maintain proper cooling.
• To maintain proper cooling, always keep all chassis panels closed and all SATA carriers installed when not being serviced.

Rack Mounting Considerations

Ambient Operating Temperature

If installed in a closed or multi-unit rack assembly, the ambient operating temperature of the rack environment may be greater than the room's ambient temperature. Therefore, consideration should be given to installing the equipment in an environment compatible with the manufacturer's maximum rated ambient temperature (TMRA).

Airflow

Equipment should be mounted into a rack so that the amount of airflow required for safe operation is not compromised.

Mechanical Loading

Equipment should be mounted into a rack so that a hazardous condition does not arise due to uneven mechanical loading.

Circuit Overloading

Consideration should be given to the connection of the equipment to the power supply circuitry and the effect that any possible overloading of circuits might have on overcurrent protection and power supply wiring. Appropriate consideration of equipment nameplate ratings should be used when addressing this concern.

Reliable Ground

A reliable ground must be maintained at all times. To ensure this, the rack itself should be grounded. Particular attention should be given to power supply connections other than the direct connections to the branch circuit (i.e. the use of power strips, etc.).

To prevent bodily injury when mounting or servicing this unit in a rack, you must take special precautions to ensure that the system remains stable. The following guidelines are provided to ensure your safety:

• This unit should be mounted at the bottom of the rack if it is the only unit in the rack.
• When mounting this unit in a partially filled rack, load the rack from the bottom to the top with the heaviest component at the bottom of the rack.
• If the rack is provided with stabilizing devices, install the stabilizers before mounting or servicing the unit in the rack.
• Slide rail mounted equipment is not to be used as a shelf or a work space.

To prevent bodily injury when mounting or servicing this unit in a rack, you must take special precautions to ensure that the system remains stable. The following guidelines are provided to ensure your safety:

• This unit should be mounted at the bottom of the rack if it is the only unit in the rack.
• When mounting this unit in a partially filled rack, load the rack from the bottom to the top with the heaviest component at the bottom of the rack.
• If the rack is provided with stabilizing devices, install the stabilizers before mounting or servicing the unit in the rack.
• Slide rail mounted equipment is not to be used as a shelf or a work space.

Warning: Do not pick up the server with the front handles. They are designed to pull the system from a rack only.

Slide rail mounted equipment is not to be used as a shelf or a work space.

2.3 Rack Mounting Instructions

This section provides information on installing the chassis into a rack unit with the rails provided. There are a variety of rack units on the market, which may mean that the assembly procedure will differ slightly from the instructions provided. You should also refer to the installation instructions that came with the rack unit you are using. Note: This rail will fit a rack between 28" and 33.5" deep.

Overview of the Rack Rails

The package includes two rail assemblies. Each is specifically designed for the left or right side of the chassis, and so marked. Each rail consists of two sections: a front section which secures to the front post of the rack and a rear section which adjusts in length and secures to the rear post of the rack.

Rear Section

Figure 2-1. Rackmount Rail (Right rail assembly shown)

Adjusting the Rail Length

Each rail assembly has a locking screw to adjust the length of the rail to fit the depth of your rack.

Installing the Rails on a Rack

1. Loosen the adjusting screw to allow the rear section to slide in the front section.
2. Push the small hooks on the front section of the rail into the holes on the front post of the rack and then down, until the spring-loaded pegs snap into the rack holes. Secure the rail to the rack with screws.
3. Pull out the rear section of the outer rail, adjusting the length until it fits within the posts of the rack and align the small hooks with the appropriate holes on the rear post of the rack. Be sure the rail is level, then mount the rear section onto the rack. Secure the rail with screws.
4. Tighten the adjusting screw.

Technical line drawing of a server rack with labeled components (no text or symbols)

Figure 2-2. Attaching the Rail Front to the Rack (Left rail shown)

Note: Figures are for illustrative purposes only. Always install servers into racks from the bottom up.

Chassis Installation

Slide the chassis into the rack so that the bottom of the chassis slides onto the bottom lip of the rails.

Technical line drawing showing a server rack with a device and a server tower, no text or symbols present

Figure 2-3. Sliding the Chassis into the Rack

Stability hazard. The rack stabilizing mechanism must be in place, or the rack must be bolted to the floor before you slide the unit out for servicing. Failure to stabilize the rack can cause the rack to tip over.

Chapter 3

Maintenance and Component Installation

This chapter provides instructions on installing and replacing main system components. To prevent compatibility issues, only use components that match the specifications and/or part numbers given.

Installation or replacement of most components require that power first be removed from the system. Please follow the procedures given in each section.

3.1 Removing Power

Before performing some setup or maintenance tasks, use the following procedure to ensure that power has been removed from the system.

Removing Power from a Node

- Use the operating system to power down the node.

Removing Power from the System

1. Use the operating system to power down all nodes.
2. Grasp the head of each power cord and gently pull it out of the back of the power supply.
3. Disconnect the cords from the power strip or wall outlet.

3.2 Accessing the System

Removing a Computing Node Drawer

Node Handle Release Latch

Figure 3-1. Removing a Node Tray

Removing a Node

1. Use the operating system to power down the node.
2. Remove any cables attached to the outside of the node.
3. Press the release latch and use the handle to pull the node from the chassis.

Removing the Chassis Cover

You can access some chassis components, such as fans, by removing the cover.

Remove two screws

Figure 3-2. Removing the Chassis Cover

Removing the Chassis Cover

The chassis top cover can be lifted off after removing two screws.

Caution: Except for short periods of time, do not operate the server without the cover in place. It provides proper airflow to prevent overheating.

3.3 Processor and Heatsink

Prepare the system for processor and heatsink installation or removal. Follow the static-sensitive device precautions when working with the processor and heatsink.

Prepare the System

1. Remove power from the system.
2. Remove the chassis cover and any components that are obstructing the CPU socket.
3. Check that the plastic protective cover is on the CPU socket and that none of the socket pins are bent. If they are, contact your retailer.
4. Refer to the Supermicro website for updates on processor and memory support.

Note: All graphics in this manual are for illustration only. Your components may look different.

ESD Precautions

Electrostatic Discharge (ESD) can damage electronic components. Handle the motherboard carefully. The following measures are generally sufficient to protect your equipment from ESD.

• Use a grounded wrist strap designed to prevent static discharge.
• Touch a grounded metal object before removing the motherboard from the antistatic bag.
• Handle the motherboard by its edges only; do not touch its components, peripheral chips, memory modules or gold contacts.
• When handling chips or modules, avoid touching their pins.
• When handling the processor, avoid touching or placing direct pressure on the LGA lands (gold contacts). Improper installation or socket misalignment can cause serious damage to the processor or the socket, and may require manufacturer repairs.
• Put the motherboard and peripherals back into their antistatic bags when not in use.
• For grounding purposes, make sure that your computer chassis provides excellent conductivity between the power supply, the case, the mounting fasteners and the motherboard.
• Use only the correct type of onboard CMOS battery. Do not install the onboard battery upside down to avoid possible explosion.

Installation Overview

After preparing the system and following ESD precautions, there are four steps to installing the processor and heatsink onto the motherboard.

1. Attach the processor to a plastic carrier to create the processor carrier assembly.
2. Attach the processor carrier assembly to the heatsink to create the processor heatsink module (PHM).
3. Prepare the socket for PHM installation.
4. Install the PHM.

Removal Overview

After preparing the system and following ESD precautions, there are three steps to removing the processor and heatsink from the motherboard.

1. Remove the PHM from the motherboard.
2. Remove the processor carrier assembly from the heatsink.
3. Remove the processor from the carrier.

Heatsink Overview

The 620BT-D Series server uses a different heatsink design for each CPU. The SNK-P0079P model is used for CPU2, the CPU closer to the mid-chassis fans. The installation and removal procedure are the same for both heatsink models.

3D CAD model of a mechanical or electronic component with labeled ports and mounting features (no readable text or symbols)

Figure 3-3. Heatsink SNK-P0078PW (for CPU1)

3D rendering of a mechanical electronic device with labeled ports and connectors (no readable text or symbols)

Figure 3-4. Heatsink SNK-P0079P (for CPU2)

Note: Thermal grease is pre-applied on new heatsinks. No additional thermal grease is needed.

Create the Processor Carrier Assembly

Process Carrier Assembly

1. Hold the processor with the gold pins (LGA lands) facing down. Locate the gold triangle at the corner of the processor and the corresponding hollowed triangle on the processor carrier as shown below. These triangles indicate the location of pin 1.

Pin 1

Processor Carrier

Pin 1

1. Turn the processor over (with the gold pins up). Locate the CPU keys on the processor and the four latches on the carrier as shown below.

Processor (Reverse Side Up) Latch Latch Carrier (Top Side Up) CPU Key Latch Latch CPU Key

1. Locate the lever on the processor socket and press it down as shown below.

Lever

1. Using pin 1 as a guide, carefully align the CPU keys on the processor (A & B) with those on the carrier (a & b) as shown below.

CPU Key (on the processor) CPU Key (on the carrier) Latch 1 2 a Latch CPU Key (on the processor) CPU Key (on the carrier)

1. Once aligned, carefully place one end of the processor under latch 1 on the carrier, and then press the other end down until it snaps into latch 2.

Processor Carrier Assembly (Top Side View) Processor Carrier Assembly

Create the Processor Heatsink Module (PHM)

If necessary, apply the proper amount of thermal grease to the underside of the heatsink.

Processor Heatsink Module

1. Turn the heatsink over with the thermal grease facing up. Pay attention to the two triangle cutouts (A, B) located at the diagonal corners of the heatsink as shown in the drawing below.
2. Hold the processor carrier assembly upside-down to locate the triangles on the processor and the carrier, which indicate pin 1.
3. Turn the processor carrier assembly over so that the gold pins are facing up. Locate the two pin 1 locations ("A" on the processor and "a" on the processor carrier assembly).
4. Align "a" on the processor carrier assembly with the triangular cutout "A" on the heatsink along with "b", "c", "d" on the processor assembly with "B", "C", "D" on the heatsink.
5. Once properly aligned, place the heatsink on the processor carrier assembly with all corners matched up, making sure that the four clips are properly securing the heatsink.

Processor Carrier Assembly Pin1 a b c D A B C Heatsink

Note: The figure is for illustrative purposes. Your components may differ slightly from the components shown.

Prepare the Socket

Remove the plastic protective cover on the socket by gently squeezing the grip tabs and pulling the cover off.

CPU Socket with Plastic Protective Cover Grip Tab

Install the PHM

After assembling the Processor Heatsink Module (PHM), you are ready to install it into the CPU socket.

1. Locate four threaded fasteners (a, b, c, d) on the CPU socket.

CPU Socket Threaded Fastener (a, b, c, d: Threaded Fasteners) CPU Socket Pin1

1. Locate four peek nuts (A, B, C, D) and four rotating wires (1, 2, 3, 4) on the heatsink as shown below.

Heatsink A, B, C, D: Peek Nut 1, 2, 3, 4: Rotating Wire a, b, c, d: Threaded Fastener Rotating Wire 4 Rotating Wire 3 Rotating Wire 1 Peek Nut CPU Socket d a Threaded Fastener Rotating Wire Peek Nut (Unlatched) (latched)

1. Check the rotating wires (1, 2, 3, 4) to make sure that they are in the unlatched position as shown.

ASSEMBL INSTRUCTIONS: 1. MOVE ALL 4 WIRE TO LOCKED POSITION 2. TORQUE 4 INITS TO 6-12MH LB

1. Align peek nut "A" (next to the triangular pin 1 on the heatsink) with threaded fastener "a" on the CPU socket. Then align peek nuts "B", "C", "D" on the heatsink with threaded fasteners "b", "c", "d" on the CPU socket, making sure that all peek nuts and threaded fasteners are properly aligned.

2. Once aligned, gently place the heatsink on the CPU socket, making sure that each peek nut is properly attached to its corresponding threaded fastener.

A, B, C, D: Peek Nut on the Heatsink B D A C b c d a, b, c, d: Threaded Fastener on the CPU socket

1. Press all four rotating wires outward and make sure that the heatsink is securely latched into the CPU socket.

Rotating Wire Rotating Wire

1. With a t30-bit screwdriver, tighten all peek nuts in the sequence of "A", "B", "C", and "D" with even pressure. To avoid damaging the processor or socket, do not use a force greater than 12 lbf-in when tightening the screws.

2. Examine all corners of the heatsink to ensure that the PHM is firmly attached to the CPU socket.

Technical illustration of a mechanical assembly with labeled components (A, B, C, D) and directional arrows indicating motion or force (no text or symbols present)

Remove the PHM from the Motherboard

Before removing the PHM from the motherboard, first shut down the system and unplug the AC power cord from all power supplies.

1. Use a t30-bit screwdriver to loosen the four peek nuts on the heatsink in the sequence of A, B, C, and D.

B D A C Peek Nut

1. Once the peek nuts have been loosened from the CPU socket, press the rotating wires inward to unlatch the PHM from the socket as shown below.

Technical illustration of a dual-chamber industrial device with cooling fans and heat sinks (no text or symbols)

1. Gently lift the PHM upward to remove it from the CPU socket.

Technical illustration of an electronic component with mounting brackets and red directional arrows indicating assembly or movement (no text or symbols present)

Remove the Carrier Assembly from the Heatsink

To remove the processor carrier assembly from the PHM, please follow the steps below:

1. Detach the four plastic clips (marked a, b, c, d) on the processor carrier assembly from the four corners of the heatsink (marked A, B, C, D) as shown below.

Processor Carrier Assembly Pin1 Pin1 a b c D A Heatsink B C

1. When all plastic clips have been detached from the heatsink, remove the processor carrier assembly from the heatsink

Technical diagram of an electronic component with red arrows indicating assembly or force direction (no text or symbols present)

Remove the Processor from the Carrier Assembly

Once you have removed the processor carrier assembly from the PHM, you are ready to remove the processor from the processor carrier by following the steps below.

1. Unlock the lever from its locked position and push it upwards to disengage the processor from the processor carrier as shown below right.

Processor Carrier Assembly Lever

1. Once the processor has been loosened from the carrier, carefully remove the processor from the carrier.

Note: Please handle the processor with care to avoid damaging it or its pins.

Isometric technical diagram of a computer motherboard with a highlighted component and red arrow indicating upward motion (no text or symbols)

3.4 Memory

The X12DPT-B6 supports up to 4TB of 3DS RDIMM DDR4 ECC memory with speeds of 3200/2933/2666 MHz in 16 memory slots and up to 4TB of Intel Optane PMem 200 Series memory with speeds of up to 3200 MHz. (See the notes below.)

Note 1: P1-DIMMC2/P2-DIMMC2 memory slots are reserved for Intel Optane PMem 200 Series only.

Note 2: Memory speed support depends on the processors used in the system.

Memory Support for the 3rd Gen Intel Xeon Scalable Processors

Memory Support for the 3rd Gen Intel Xeon Scalable Processors
Type	Ranks Per DIMM & Data Width	DIMM Capacity (GB)		Speed (MT/s); Voltage (V); Slot Per Channel (SPC) and DIMM Per Channel (DPC)
				1DPC (1-DIMM Per Channel)	2DPC (2-DIMM Per Channel)
		8Gb 16Gb	1.2 V 1.2 V
RDIMM	SRx8 8GB 16GB			3200 2933*
	SRx4 16GB 32GB
	DRx8 16GB 32GB
	DRx4 32GB 64GB
RDIMM-3DS (4R/8R)X4		2H-64GB4H-128GB	2H-128GB4H-256GB

Memory Population Table for the 3rd Gen Intel Scalable Processor

Memory Population for the X12 DP Motherboard, 20 DIMM Slots
CPUs/DIMMs Memory Population Sequence
1 CPU & 1 DIMM	A1
1 CPU & 2 DIMMs*	A1, E1
1 CPU & 4 DIMMs*	A1, E1, C1, G1
1 CPU & 6 DIMM	A1, E1, C1, G1, B1, F1
1 CPU & 8 DIMMs*	A1, E1, C1, G1, B1, F1, D1, H1
1 CPU & 9 DIMMs	A1, E1, C1, G1, B1, F1, D1, H1, C2 (C2: Reserved for PMem 200 Series only)
2 CPUs & 2 DIMMs*	CPU1: A1CPU2: A1
2 CPUs & 4 DIMMs*	CPU1: A1, E1CPU2: A1, E1
2 CPUs & 6 DIMMs	CPU1: A1, E1, C1, G1CPU2: A1, E1
2 CPUs & 8 DIMMs*	CPU1: A1, E1, C1, G1CPU2: A1, E1, C1, G1
2 CPUs & 10 DIMMs	CPU1: A1, E1, C1, G1, B1, F1CPU2: A1, E1, C1, G1
2 CPUs & 12 DIMMs*	CPU1: A1, E1, C1, G1, B1, F1CPU2: A1, E1, C1, G1, B1, F1
2 CPUs & 14 DIMMs	CPU1: A1, E1, C1, G1, B1, F1, D1, H1CPU2: A1, E1, C1, G1, B1, F1
2 CPUs & 16 DIMMs*	CPU1: A1, E1, C1, D1, B1, F1, G1, H1CPU2: A1, E1, C1, D1, B1, F1, G1, H1
2 CPUs & 18 DIMMs	CPU1: A1, E1, C1, D1, B1, F1, G1, H1, C2 (C2: Reserved for PMem 200 Series only)CPU2: A1, E1, C1, D1, B1, F1, G1, H1, C2 (C2: Reserved for PMem 200 Series only)

Note 1: P1-DIMMC2 and P2-DIMMC2 are reserved for Intel® Optane™ PMem 200 Series only.

Note 2: To maximize memory performance, please use the memory configurations marked with "*" above (also shaded in orange) as these configurations are recommended by Supermicro for optimal memory performance.

Intel Optane PMem 200 Series Memory Population Table

Note: PMem 200 Series are supported on 3rd gen Intel Xeon Scalable Platinum, Gold and selected Silver processors.

20-DIMM Motherboard PMem Population within 1 CPU socket

DDR4+Pmem

Mode

AD Inter-leave

P1-DIMMF1

P1-DIMME1

P1-DIMMH1

P1-DIMMG1

P1-DIMMG2

P1-DIMMC2

P1-DIMMC1

P1-DIMMD1

P1-DIMMA1

P1-DIMMB1

4+4

ADMM

One - x4

DDR4

PMem

DDR4

PMem

-

-

PMem

DDR4

PMem

DDR4

6+1

AD

One - x1

DDR4

DDR4

-

DDR4

-

-

DDR4

PMem

DDR4

DDR4

DDR4

DDR4

PMem

DDR4

-

-

DDR4

-

DDR4

DDR4

PMem

DDR4

DDR4

DDR4

-

-

DDR4

DDR4

DDR4

-

DDR4

DDR4

DDR4

-

-

-

PMem

DDR4

DDR4

DDR4

DDR4

-

DDR4

DDR4

-

-

DDR4

DDR4

PMem

DDR4

DDR4

DDR4

DDR4

PMem

-

-

-

DDR4

DDR4

DDR4

DDR4

PMem

DDR4

DDR4

-

-

DDR4

DDR4

-

DDR4

8+1

AD

One - x1

DDR4

DDR4

DDR4

DDR4

-

PMem

DDR4

DDR4

DDR4

DDR4

Legend (for the table above)
DDR4 Type and Capacity
DDR4	See Validation Matrix (DDR4 DIMMs validated with DCPMM)
Capacity
PMem Any Capacity (Uniformly for all channels for a given configuration)

• Mode definitions: AD = App Direct Mode, MM = Memory Mode.
- For MM, NM/FM ratio is between 1:4 and 1:16. The capacity not used for FM can be used for AD. (NM = Near Memory; FM = Far Memory).
- Matrix targets configs for optimized PMem to DRAM cache ratio in MM mode.
- For each individual population, different PMem rearrangements among channels are permitted so long as the configuration doesn't break X12 DP Memory population rules.
- Ensure the same DDR4 DIMM type and capacity are used for each DDR4 + PMem population.
- If the system detects an unvalidated configuration, then the system issues a BIOS warning. The CLI functionality is limited in non-POR configurations, and select commands will not be supported.

Validation Matrix (DDR4 DIMMS w/PMem 200 Series)
DIMM Type	Ranks Per DIMM & Data Width (Stack)	DIMM Capacity (GB)
		DRAM Density
		8Gb 16Gb
RDIMM (up to 3200)	1Rx8 N/A N/A
	1Rx4 16GB 32GB
	1Rx8 16GB 32GB
	1Rx4 32GB 64GB
RDIMM 3DS (up to 3200)	4Rx4 (2H) N/A 128GB
	8Rx4 (4H) NA 256GB

U/D_LED1 JPFR3 JPFR2 JPMIE1 JUDB0 JUSB1 JVSA1 VGA COM11 COM27 JNCSI JRP1 JRP1L1 JTPM1 TPM/PORT80 S-SATA 2-5 S-CATA 2-5 SLOT1 BT1 CE SUPER X12DPT-B6 REV:1.00 BIOS LICENSE FAN3 PCH SIOTAN CMOS CLEAR PCH P1-DIMMC2 P1-DIMMC1 P1-DIMMD1 P1-DIMMA1 P1-DIMMB1 CPU1 CPU2 CPU1 CPU2 CPU1 CPU2 CPU1 CPU2 CPU1 CPU2 CPU1 CPU2 CPU1 CPU2 CPU1 CPU2 CPU1 CPU2 CPU1 CPU2 CPU1 CPU2 CPU1 CPU2 CPU1 CPU2 CPU1 CPU2 CPU1 CPU2 CPU1 CPU2 CPU1 CPU2 CPU1

Figure 3-5. DIMM Slots

DIMM Installation

1. Insert the desired number of DIMM modules based on the recommended DIMM population table on page 45.
2. Align the DIMM module key with the receptive point on the single-latch DIMM slot.

Receptive Point

1. Push the release tab outwards to unlock the slot.

Notch Release Tab

Technical schematic diagram of a device layout with labeled components and connections

1. Press both ends of the module straight down into the slot until the module snaps into place.

Push both ends straight down into the memory slot.

1. Push the release tab to the lock position to secure the module into the slot.

Technical line drawing of a mechanical component with a blue circular arrow indicating rotation (no text or symbols)

DIMM Removal

Reverse the steps above to remove the DIMM modules from the motherboard.

3.5 Motherboard Battery

The motherboard uses non-volatile memory to retain system information when system power is removed. This memory is powered by a lithium battery residing on the motherboard.

Replacing the Battery

Begin by removing the node from the system.

1. Push aside the small clamp that covers the edge of the battery. When the battery is released, lift it out of the holder.
2. To insert a new battery, slide one edge under the lip of the holder with the positive (+) side facing up. Then push the other side down until the clamp snaps over it.

Note: Handle used batteries carefully. Do not damage the battery in any way; a damaged battery may release hazardous materials into the environment. Do not discard a used battery in the garbage or a public landfill. Please comply with the regulations set up by your local hazardous waste management agency to dispose of your used battery properly.

LITHIUM BATTERY BATTERY HOLDER

Figure 3-6. Installing the Onboard Battery

Warning: There is a danger of explosion if the onboard battery is installed upside down (which reverses its polarities). This battery must be replaced only with the same or an equivalent type recommended by the manufacturer (BR2032).

3.6 Storage Drives

The CSE-827BD chassis supports 12 hot-swap 3.5" storage drives (HDDs or SSDs). Each node controls six drives.

• DNTR: 6 hybrid SATA3/NVMe drives per node
- DNC8R: 6 hybrid SAS3/SATA3/NVMe drives (SAS3 supported via Broadcom 3808) per node

Drive Carriers

Each drive carrier has two LED indicators: an activity indicator and a status indicator. For RAID configurations using a controller, the meaning of the status indicator is described in the table below. For OS RAID or non-RAID configurations, some LED indications are not supported, such as hot spare. For VROC configurations, refer to the VROC appendix in this manual.

Drive Carrier LED Indicator
	Color Blinking Pattern Behavior for Device
Activity LED	Blue Solid On	SAS/NVMe drive installed
	Blue Blinking	I/O activity
Status LED	Red Solid On	Failure of drive with RSTe support
	Red Blinking	at 1 Hz Rebuild drive with RSTe support
	Red	Blinking with two blinks and one stop at 1 Hz	Hot spare for drive with RSTe support (not supported in VMD mode)
	Red On for five seconds, then off Power on for drive with RSTe support
	Red Blinking	at 4 Hz Identify drive with RSTe support
	Green Solid On		Safe to remove NVMe device (not supported in VMD mode)
	Amber Blinking	at 1 Hz	Attention state---do not remove NVMe device (not supported in VMD mode)

Note: Enterprise level hard disk drives are recommended for use in Supermicro chassis and servers. For information on recommended HDDs, see the Supermicro website, http://www.supermicro.com/products/nfo/files/storage/SBB-HDDCompList.pdf.

Drive Configuration

The CSE-827BD chassis contains two separate computing node drawers, each with its own motherboard. Each node controls a set of six drives. If a node drawer is pulled out of the chassis, the drives associated with that node will power down.

Node Drawer Locations
Node A controls drives A1-A6	Node B controls drives B1-B6

Technical diagram of a server rack with labeled components A and B, showing internal structure and ventilation slots.

Figure 3-7. Storage Drives and the Corresponding Nodes

Installing Drives

Removing Drive Carriers from the Chassis

1. Press the release button on the drive carrier. This extends the drive carrier handle.
2. Use the handle to pull the carrier out of the chassis (Figure 3-7).
3. Remove the dummy drive from the carrier (Figure 3-8).

Caution: Except for short periods of time (swapping drives), do not operate the server with the drive carriers removed from the bays, regardless of how many drives are installed, for proper airflow.

Technical diagram showing server rack and internal drive components with labeled parts 1 and 2

Figure 3-8. Removing a Drive Carrier

Technical line drawing of a computer drive chassis showing internal components and mounting points (no text or labels)

Figure 3-9. Removing Dummy Drive from the Drive Carrier

Installing a Drive

1. Install a new drive into the carrier with the printed circuit board side facing down so that the mounting holes in the drive align with those in the carrier.

2. Secure the hard drive into the carrier with the screws.

Technical diagram showing assembly of a hard drive into a plastic housing, with labeled components and directional arrow.

Figure 3-10. Installing Hard Drive

1. Insert the drive and carrier into its bay with the release button on the right. When the carrier reaches the rear of the bay, the release handle starts to retract.

2. Push the upper part of the drive carrier handle until it clicks into the locked position.

Hot-Swap for NVMe Drives

Supermicro servers support NVMe surprise hot-swap. For even better data security, NVMe orderly hot-swap is recommended. NVMe drives can be ejected and replaced remotely using BMC.

Note: If you are using VROC, see the Hot-Swap Drives section in Appendix C instead.

Ejecting a Drive

1. BMC > System > Storage Monitoring > Physical View
2. Select Device, Group and Slot, and click Eject. After ejecting, the drive Status LED indicator turns green.
3. Remove the drive.

Note that Device and Group are categorized by the CPLD design architecture. The 620BT-D Series server has one Device and one Group

Slot is the slot number on which the NVMe drives are mounted.

Product Dashboard System Component info Health Event Log Storage Monitoring Configuration Remote Control Maintenance Overview Physical View Logical View Controller Physical View Blank Unblink Eject Insert Shift LED Status Supported Actions Disk Info4 Capacity Link Speed Connected Logical Drive Connected Controller 0.0 SAMSUNG MZCUWROGIMU- 00003 N/A 8.0 GTI- N/A N/A White Device5 0.1 SAMSUNG MZCUWROGIMU- 00003 N/A 8.0 GTI- N/A N/A White Device5

Figure 3-11. BMC Screenshot

Replacing the Drive

1. Insert the replacement drive.
2. BMC > System > Storage Monitoring > Physical View
3. Select Device, Group and Slot and click Insert. The drive Status LED indicator flashes red, then turns off. The Activity LED turns blue.

Checking the Temperature of an NVMe Drive

There are two ways to check using BMC.

Checking a Drive

• BMC > Server Health > NVMe SSD – Shows the temperatures of all NVMe drives, as in Figure 3-11.
• BMC > Server Health > Sensor Reading > NVME_SSD – Shows the single highest temperature among all the NVMe drives.

Adding an Expansion Card

The CSE-827BD chassis supports expansion cards. Riser cards are used to mount the expansion cards. Each node can accommodate three PCIe cards.

Slot for M.2 Carrier-and-Riser Card Expansion Card Screw PCIe 4.0 x8 Slot Shields

Figure 3-12. Expansion Card Slots

Installing PCIe 4.0 x8 Expansion Cards

1. If necessary, power down the node, remove it from the chassis, remove the expansion card screw (See Figure 3-12) and set aside.
2. Remove the PCI slot shields
3. If the riser card (p/n RSC-P2R-88G4) is not installed, attach the riser card to the riser card bracket.
4. Insert an expansion card into the riser slot to create an assembly.
5. Align the assembly with SLOT 1 on the motherboard and the PCI slot shield at the chassis rear.
6. Insert the assembly into the motherboard and install a screw to hold the expansion card to the chassis.
7. Continue to install the M.2 carrier-and-riser card or reinstall the expansion card screw, reinsert the node into the chassis and power up the system.

Riser Card Bracket Expander Card Riser Card PCI Slot

Figure 3-13. Installing Expansion Card
Note: Figure is for illustrative purposes only. Your system may or may not resemble this picture.

Installing M.2 Carrier-and-Riser Card

1. Power down the node and remove it from the chassis.

2. Remove the expansion card screw (See Figure 3-12) and set aside. Remove the PCI slot shield.

3. Attach brackets onto the M.2 carrier-and-riser card, if necessary.

4. Insert an expansion card into the riser slot of the M.2 carrier-and-riser card to create an assembly (not shown).

5. Install the M.2 SSD, if necessary.

Bracket Bracket

Figure 3-14. Installing Brackets

1. Align the assembly with SLOT2 on the motherboard and the PCI slot shield at the node rear and insert the assembly into the motherboard.

2. Reinstall the expansion card screw, reinsert the node into the chassis, and power up the system.

M.2 SSD Riser Slot PCI Slot Shield

Figure 3-15. Installing Expansion Card onto M.2 Carrier-and-Riser Card

AIOM Card

The Supermicro Advanced Input/Output module (AIOM) card provides options for network connection. It is inserted into an AIOM slot on the motherboard tray.

Removing the AIOM Card

1. Press the release tab and loosen the thumbscrew on the AIOM card.

2. Grasp the release tab and the thumbscrew and pull the AIOM out of the node tray.

Thumbscrew Release Tab

Figure 3-16. AIOM Card Position on Node Drawer Rear

Installing the AIOM Card

1. Insert the AIOM card into the motherboard tray slot as shown until the release tab retracts.

2. Tighten the thumbscrew.

3.7 System Cooling

Fans

Fan speed is controlled by a system temperature setting in BMC. If a fan fails, the remaining fans will ramp up to full speed. The system can continue to run with a failed fan. Replace any failed fan at your earliest convenience with the same type and model.

Technical line drawing of a server rack with two fans and green arrows indicating mounting points (no text or symbols present)

Figure 3-17. System Fan Placement

Changing a System Fan

1. Determine which fan is failing. If possible, use BMC. If not, remove the chassis cover while the power is on, and examine the fans to determine which one has failed.
2. Remove power from the system as described in Section 3.1.
3. Remove the fan cable from the backplane for the failed fan and the adjacent fan.
4. Lift the fan housing up and out of the chassis.
5. Push the fan up from the bottom and out of the top of the housing.

HDTV HDTV

Figure 3-18. Replacing a System Fan in the Fan Housing

1. Place the replacement fan into the vacant space in the housing while making sure the arrows on the top of the fan (indicating air direction) point in the same direction as the arrows on the other fans.
2. Put the fan housing back into the chassis and reconnect the cable.
3. Power on the system to confirm that the fan is working properly before replacing the chassis cover.

Installing Air Shroud

The system requires air shroud for each node to maximize airflow efficiency.

Installing Air Shroud

The motherboard, any expansion cards, and all components must be installed in the node tray. Place the air shroud as pictured and secure with screws.

Technical line drawing of an internal server rack with visible components and ventilation ducts (no text or labels)

Figure 3-19. Installing Air Shroud

Checking the Airflow

Checking Airflow

• Make sure there are no objects obstructing the airflow in and out of the chassis.
- Except for brief periods while swapping hard drives, do not operate the server without the drive carriers in the drive bays.
- Make sure no wires or foreign objects obstruct airflow through the chassis. Pull all excess cabling out of the airflow path or use shorter cables.

3.8 Power Supply

The chassis features redundant power supplies. The power modules can be changed without powering down the system. New units can be ordered directly from Supermicro or authorized distributors.

These power supplies are auto-switching capable. This feature enables them to automatically sense the input voltage and operate at a 100-120v or 180-240v. An amber light will be illuminated on the power supply when the power is off. An illuminated green light indicates that the power supply is operating.

Replacing the Power Supply

1. Unplug the AC cord from the module to be replaced.
2. Push the release tab on the back of the power supply as illustrated.

2 Release Tab 3

Figure 3-20. Power Supply Release Tab

1. Pull the power supply out using the handle provided.
2. Replace the failed power module with the same model.
3. Push the new power supply module into the power bay until it clicks.
4. Plug the AC power cord back into the module.

3.9 Cable Routing Diagram

Refer to the diagram below for a representation of how the storage cables are routed through the node. When disconnecting cables to add or replace components, refer to this diagram so you can reroute them in the same manner.

SATA Cable CBL-SAST-1243-100 SAS Cable CBL-SAST-1221-100

Online Cable Matrix

Interior view of a server rack with multiple CPU racks and memory modules (no visible text or labels)

Figure 3-21. Cable Routing Diagram

3.10 BMC Reset

The BMC can be reset using the UID button.

• Reset – Press and hold the button. After six seconds, the LED blinks at 2Hz. The BMC resets and the reset duration is \~250 ms. Then the BMC starts to boot.
• Restore factory default configuration – Hold the button for twelve seconds. The LED blinks at 4Hz while the defaults are configured. Note: All BMC settings including username and password will be removed except the FRU and network settings.

Firmware update – When the BMC firmware is being updated, the UID LED blinks at 10Hz.

BMC Reset Options
Event UID LED	BMC Heartbeat LED
Reset Blue, Blinks at 2Hz Green, solid
Restore Defaults	Blue, Blinks at 4Hz Off
Update Blue, Blinks at 10Hz

Chapter 4

Motherboard Connections

This section describes the connections on the motherboard and provides pinout definitions. Note that depending on how the system is configured, not all connections are required.

The LEDs on the motherboard are also described here. A motherboard layout indicating component locations may be found in Chapter 1. More detail can be found in the Motherboard Manual.

Please review the Safety Precautions in Appendix A before installing or removing components.

4.1 Power Connections

Power Supply Connectors

There are four 6-pin 12V DC power connectors (JPWR1/JPWR2/JPWR3/JPWR4) on the motherboard to provide adequate power supply to your system.

12V 6-pin Power Pin Definitions

Pin# Definition

1 - 3 Ground

4 - 6 +12V

4.2 Headers and Connectors

Fan Headers

There are four 4-pin fan headers on the motherboard: two (FAN3\~FAN4) on the front plane (see locations below), and two (FAN1\~FAN2) on the HDD backplane. All these 4-pin fan headers are backwards compatible with the traditional 3-pin fans. However, fan speed control is available for 4-pin fans only by Thermal Management via the BMC interface. Refer to the table below for pin definitions.

Fan HeaderPin Definitions

Pin# Definition

1 Ground

2 2.5A/+12V

3 Tachometer

4 PWM_Control

VROC RAID Key Header

A VROC RAID Key header is located at JRK1 on the motherboard. Install a VROC RAID Key on JRK1 for NVMe RAID support as shown in the illustration below. Please refer to the layout below for the location of JRK1.

Intel VROC KeyPin Definitions
Pin#	Definition
1	Ground
2	3.3V Standby
3	Ground
4	PCH RAID Key

VROC Key VROC Key Header (JRK1)

Note: The graphics contained in this user's manual are for illustration only. The components installed in your system may or may not look exactly the same as the graphics shown in the manual.

TPM/Port 80 Header

The JTPM1 header is used to connect a Trusted Platform Module (TPM)/Port 80, which is available from Supermicro (optional). A TPM/Port 80 connector is a security device that supports encryption and authentication in hard drives. It allows the motherboard to deny access if the TPM associated with the hard drive is not installed in the system. See the layout below for the location of the TPM header. Please go to the following link for more information on the TPM: http://www.supermicro.com/manuals/other/TPM.pdf.

Trusted Platform Module HeaderPin Definitions
Pin# Definition Pin# Definition
1 +3.3 V 2 SPI_CS#
3 RESET# 4 SPI_MISO
5 SPI_CLK 6 GND
7 SPI_MOSI 8 NC
9 +3.3 V Stdby 10 SPI_IRQ#

NCSI Connector

The NCSI header (JNCSI) is used to connect a Network Interface Card (NIC) to the motherboard so that the BMC is able to poll the temperature reading from it.

Note: For detailed instructions on how to configure Network Interface Card (NIC) settings, please refer to the Network Interface Card Configuration User's Guide posted on the web page under the link: http://www.supermicro.com/manuals/.

I-SATA 3.0 and S-SATA 3.0 Ports

The X12DPT-B6 has eight I-SATA 3.0 ports (I-SATA0\~7) and six S-SATA ports (S-SATA0-1, S-SATA2\~5). These SATA ports, supported by the C621A chipset, provide serial-link signal connections.

4.3 Input/Output Ports

See the figure below for the locations and descriptions of the I/O ports on the rear of the motherboard.

Close-up of a network device with labeled ports and connectors (no text or symbols visible)

Figure 4-1. I/O Port Locations and Definitions

Rear I/O Ports
# Description # Description
1 VGA Port 3 Dedicated BMCLAN
2 USB0/1 (3.0) 4 UID Switch

VGA Connection

The VGA port is located at JVGA1 on the front I/O panel. The VGA connection provides analog interface support between the computer and the video displays. Refer to the layout below for the location of the VGA connection.

COM Port

The COM (communication) port (COM1) supports serial link interface. Refer to the layout below for the location of the COM port.

BMC LAN Port

The BMC dedicated LAN (BMC_LAN1), provides LAN support for the BMC (Baseboard Management Controller). Please refer to the LED Indicator section for LAN LED information.

Universal Serial Bus (USB) Ports and Headers

An 18-pin USB connector, located on the rear I/O panel, supports two USB 3.0 ports (USB0/1) via USB cables.

Rear I/O Panel USB0/1 (3.0)Pin Definitions
Pin# Definition Pin# Definition
A1 VB	US B1 Power
A2 D-	B2 USB_N
A3 D+	B3 USB_P
A4 GND	B4 GND
A5 Stca	SSRX- B5 USB3_RN
A6 Stca	SSRX+ B6 USB3_RP
A7 GND	B7 GND
A8 Stca	SSTX- B8 USB3_TN
A9 Stca	SSTX+ B9 USB3_TP

UID (Unit Identification)/BMC Reset Switch and UID/BMC Reset LED Indicators

A UID LED/BMC Reset switch (JUIDB1) is located on the rear side of the motherboard. This switch has dual functions. It can be used to identify a system unit that is in need of service, and it can also be used to reset the BMC settings.

When functioning as a UID LED switch, it can turn the UID LED (UID_LED1) on and off to identify a unit that may require service.

When functioning as a BMC reset switch and working in conjunction with BMC Heartbeat LED (LED2), JUIDB1 will trigger a cold reboot when the user presses and holds the switch for 6 seconds. It will also restore the BMC to the manufacturer's default when the user presses and holds the switch for 12 seconds.

To achieve these dual purposes, the UID LED/BMC Reset switch works in conjunction with the BMC Heartbeat LED (LED2). Refer to the tables below for more details. Please note that UID can also be triggered via BMC on the motherboard. Refer to the BMC User's Guide posted on our website at http://www.supermicro.com for more information on BMC.

UID/BMC Reset Switch (JUIDB1) Features & Settings
When Used as a UID LED Switch (works with UID_LED1)	When Used as a BMC Reset Switch (works with BMC Heartbeat LED (LED2))
Color Status	BMC Heartbeat LED LED2 Green	Blinking: BMC Normal
Blue: On Unit Identified	BMC Reset: Press & hold the switch (JUIDB1) for 6 seconds	LED2: Solid green: during reboot
Press the switch (JUIDB1) to turn on and off the UID LED.		Triggering a cold reboot; LED: solid green on during cold reboot
	BMC Reset: Press & hold the switch (JUIDB1) for 12 seconds	LED2: Solid green: during BMC reset
		BMC: Reset to the manufacturer's default; LED: solid green on during BMC Reset

UID/BMC Reset Switch (JUIDB1) Pin Definitions
Pin#	Definition
1	Ground
2	Ground
3	Button In
4	Button In

4.4 Jumpers

How Jumpers Work

To modify the operation of the motherboard, jumpers can be used to choose between optional settings. Jumpers create shorts between two pins to change the function of the connector. Pin 1 is identified with a square solder pad on the printed circuit board. Refer to the diagram below for an example of jumping pins 1 and 2. Refer to the motherboard layout page for jumper locations.

Note: On two-pin jumpers, "Closed" means the jumper is on, and "Open" means the jumper is off the pins.

graph TD
    A["Connector Pins"] --> B["Top View"]
    B --> C["Connector Pins with a Jumper installed on Pins 1 and 2"]

CMOS Clear

JBT1 is used to clear CMOS, which will also clear any passwords. Instead of pins, this jumper consists of contact pads to prevent accidentally clearing the contents of CMOS.

To Clear CMOS

1. First power down the system and unplug the power cord(s).
2. Remove the cover of the chassis to access the motherboard and remove the battery from the motherboard.
3. Short the CMOS pads with a metal object such as a small screwdriver for at least four seconds.
4. Remove the screwdriver (or shorting device).
5. Re-install the battery.
6. Replace the cover, reconnect the power cord(s), and power on the system.

Note 1: Clearing CMOS will also clear all passwords.

Note 2: Do not use the PW_ON connector to clear CMOS.

ME Recovery

JPME1 is used for ME Firmware Recovery mode, which will limit system resource for essential function use only without putting restrictions on power use. In the single operation mode, online upgrade will be available via Recovery mode. See the table below for pin definitions.

ME RecoveryJumper Settings

Jumper Setting Definition

Pins 1-2 Normal (Default)

Pins 2-3 ME Recovery

4.5 LED Indicators

BMC LAN LEDs

A BMC-dedicated LAN (BMC_LAN1) is supported by the onboard Baseboard Management controller. The amber LED on the right indicates activity, while the LED on the left indicates the speed of the connection. Refer to the table below for more information.

BMC LAN LEDs
Color/State Definition
Link (Left)	Green: SolidAmber: Solid	100 Mbps1Gbps
Activity (Right) Amber: Blinking Active

BMC LAN

Activity LEDLink LED

Unit ID LED

A rear UID LED indicator (UID_LED1) is located next to the UID switch on the motherboard. This UID indicator provides easy identification of a system unit that may need service.

UID LEDLED Indicator
LED Color Definition
Blue: On Unit	Identified

BMC Heartbeat LED

A BMC Heartbeat LED is located at LED2 on the motherboard. When LED2 is blinking, the BMC is functioning normally. Refer to the table below for more information.

BMC Heartbeat LED Indicator
LED Color Definition
Green:Blinking	BMC Normal

Chapter 5

Software

After the hardware has been installed, you can install the Operating System (OS), configure RAID settings and install the drivers.

5.1 Microsoft Windows OS Installation

If you will be using RAID, you must configure RAID settings before installing the Windows OS and the RAID driver. Refer to the RAID Configuration User Guides posted on our website at www.supermicro.com/support/manuals.

Installing the OS

1. Create a method to access the MS Windows installation ISO file. That might be a DVD, perhaps using an external USB/SATA DVD drive, or a USB flash drive, or the BMC KVM console.
2. Retrieve the proper RST/RSTe driver. Go to the Supermicro web page for your motherboard and click on "Download the Latest Drivers and Utilities", select the proper driver, and copy it to a USB flash drive.
3. Boot from a bootable device with Windows OS installation. You can see a bootable device list by pressing F11 during the system startup.

Please select boot device: ATEN Virtual CDROM YSOJ → IPMI virtual drive (Legacy) ASUS SDRW-08D2S-U F601 → USB DVD device (Legacy) USB FLASH DRIVE PMAP → USB flash drive with OS installation (Legacy) IBA 40-10G Slot 1900 v1060 → PXE boot (Legacy) UEFI: ATEN Virtual CDROM YSOJ → IPMI virtual drive (UEFI) UEFI: ASUS SDRW-08D2S-U F601 → USB DVD device (UEFI) UEFI: Built-in EFI Shell Enter Setup ↑ and ↓ to move selection ENTER to select boot device ESC to boot using defaults

Figure 5-1. Select Boot Device

1. During Windows Setup, continue to the dialog where you select the drives on which to install Windows. If the disk you want to use is not listed, click on "Load driver" link at the bottom left corner.

Where do you want to install Windows? Name Total size Free space Type Refresh Delete Format New Load driver Extend We couldn't find any drives. To get a storage driver, click Load driver. Next

Figure 5-2. Load Driver Link

To load the driver, browse the USB flash drive for the proper driver files.

• For RAID, choose the SATA/sSATA RAID driver indicated then choose the storage drive on which you want to install it.

• For non-RAID, choose the SATA/sSATA AHCI driver indicated then choose the storage drive on which you want to install it.

• Once all devices are specified, continue with the installation.

• After the Windows OS installation has completed, the system will automatically reboot multiple times.

5.2 Driver Installation

The Supermicro website contains drivers and utilities for your system at https://www.supermicro.com/wdl/driver. Some of these must be installed, such as the chipset driver.

After accessing the website, go into the CDR_Images (in the parent directory of the above link) and locate the ISO file for your motherboard. Download this file to a USB flash drive or a DVD. (You may also use a utility to extract the ISO file if preferred.)

Another option is to go to the Supermicro website at http://www.supermicro.com/products/. Find the product page for your motherboard, and "Download the Latest Drivers and Utilities". Insert the flash drive or disk and the screenshot shown below should appear.

SUPERMICRO X12DPT-B6 Motherboard Drivers & Tools (Win2019) Intel C621A Chipset X12DPT-B6(S) SUPERMICRO Computer Inc. Intel Chipset INF files Microsoft .Net Framework 4.8 (Optional) ASPEED Graphics Driver Intel Virtual RAID on CPU Intel PRO Network Connections Drivers SUPERMICRO SuperDoctor 5 Build driver diskettes and manuals Browse CD Auto Start Up Next Time For more information, please visit SUPERMICRO's web site.

Figure 5-3. Driver & Tool Installation Screen

Note: Click the icons showing a hand writing on paper to view the readme files for each item. Click the computer icons to the right of these items to install each item (from top to the bottom) one at a time. After installing each item, you must re-boot the system before moving on to the next item on the list. The bottom icon with a CD on it allows you to view the entire contents.

5.3 SuperDoctor® 5

The Supermicro SuperDoctor 5 is a program that functions in a command-line or web-based interface for Windows and Linux operating systems. The program monitors such system health information as CPU temperature, system voltages, system power consumption, fan speed, and provides alerts via email or Simple Network Management Protocol (SNMP).

SuperDoctor 5 comes in local and remote management versions and can be used with Nagios to maximize your system monitoring needs. With SuperDoctor 5 Management Server (SSM Server), you can remotely control power on/off and reset chassis intrusion for multiple systems with SuperDoctor 5 or BMC. SuperDoctor 5 Management Server monitors HTTP, FTP, and SMTP services to optimize the efficiency of your operation.

SuperDoctor® Manual and Resources

SuperDoctor 5 Health Info System Info Configuration Export Power Control Select Language: English [admin] Login Motherboard: X8DTU-LN4+ Fan Speed FAN 5 FAN 8 Voltage CPU1 Vcc CPU2 DIMM -1.1 V -3 V -5VSB -12 V PS Status Temperature 100 -210 20 -46 30 -126 40 -40 100 -210 20 -46 30 -126 40 -40 100 -210 20 -46 30 -126 40 -40

Figure 5-4. SuperDoctor 5 Interface Display Screen (Health Information)

5.4 BMC

The motherboard provides remote access, monitoring and management through the baseboard management controller (BMC) and other management controllers distributed among different system modules. There are several BIOS settings that are related to BMC. For general documentation and information on BMC, visit our website at:

www.supermicro.com/en/solutions/management-software/bmc-resources

BMC ADMIN User Password

For security, each system is assigned a unique default BMC password for the ADMIN user. This can be found on a sticker on the chassis and a sticker on the motherboard. The sticker also displays the BMC MAC address.

BMC AC1F6BC PWD SUOKJ

Figure 5-5. BMC Password Label

See the motherboard layout in Chapter 1 for the location of the label.

Chapter 6

Optional Components

This chapter describes optional system components and installation procedures.

6.1 Optional Parts List

Optional Parts List
Description Part Number Quantity
M.2 NVMe HW RAID Controller Card AOC-SMG3-2M2-B	1

6.2 M.2 NVMe Carrier Card

See Chapter 3 Installing M.2 Carrier Card for steps to install the M.2 carrier card and SSDs.

6.3 GPU Support

The 620BT-DNTR and 620BT-DNC8R support up to two NVidia T4 GPUs per node. See Chapter 3 Installing Expansion Cards for steps to install the NVidia T4 GPUs.

Chapter 7

Troubleshooting and Support

7.1 Information Resources

Website

A great deal of information is available on the Supermicro website, supermicro.com.

SUPERMICRO TECHNOLOGY & OUR PLANET Supermicro Resource-Saving Systems: Reduce Cost and Environmental Impact

Figure 7-1. Supermicro Website

• Specifications for servers and other hardware are available by clicking the menu icon, then selecting the Products option.
• The Support option offers downloads (manuals, BIOS/BMC, drivers, etc.), FAQs, RMA, warranty, and other service extensions.

Direct Links for the 620BT-D Series System

620BT-D Series specifications page

X12DPT-B6 motherboard page for links to the Quick Reference Guide, User Manual, validated storage drives, etc.

BPN-NVME4-217BHQ-S6 Backplane Manual

Direct Links for General Support and Information

Frequently Asked Questions

Add-on card descriptions

TPM User Guide

General Memory Configuration Guide: X12

Direct Links (continued)

BMC User Guide

SuperDoctor5 Large Deployment Guide

For validated memory, use our Product Resources page

Product Matrices page for links to tables summarizing specs for systems, motherboards, power supplies, riser cards, add-on cards, etc.

Security Center for recent security notices

Supermicro Phone and Addresses

7.2 Baseboard Management Controller (BMC)

The system supports the Baseboard Management Controller (BMC). BMC is used to provide remote access, monitoring and management. There are several BIOS settings that are related to BMC.

For general documentation and information on BMC, please visit our website at: https://www.supermicro.com/en/solutions/management-software/bmc-resources.

19! Welcome back! System Component Info Health Event Log Storage Monitoring Configuration + Remote Control Maintenance + System Firmware Version 9.10.04 BETA Firmware Build Time 04/05/2020 Redfish Version 1.8.0 BOS Version BOS Build Time CPLD Version F0.05.05 BMC Mac Address AG:1F:BB:3D:22.6C LAN 1 Mac Address AC:1F:BB:CA:22.8A LAN 2 Mac Address AC:1F:BB:CA:22.8B LAN 1 Mac Address 9C:C4:7A:D8:SE:7E LAN 2 Mac Address 9C:C4:7A:D8:SE:7F LAN 1 Mac Address AC:1F:BB:CF:37.FA LAN 2 Mac Address AC:1F:BB:CF:37.FB Storage USB Control Firmware Update Sensor Readings Host Server Host Name Server IP Address 172.31.58.213 IPv6 Address 1 le80.0.0.0 ac11/buff feid 3d43 IPv6 Address 2 IPv6 Address 3 IPv6 Address 4 IPv6 Address 5 IPv6 Address 6 Power Consumption Min Peak Average Usage Max Peak X:Time(min), Y:Power Consumption (Watt) Remote Console Preview JAVA plug-in HTTPS released image

Figure 7-2. BMC Sample

7.3 Troubleshooting Procedures

Use the following procedures to troubleshoot your system. If you have followed all of the procedures below and still need assistance, refer to the Technical Support Procedures or Returning Merchandise for Service section(s) in this chapter. Power down the system before changing any non hot-swap hardware components.

No Power

1. Check that the power LED on the motherboard is on.
2. Make sure that the power connector is connected to your power supply.

UO LED JPM1 JPFR3 JPME1 S SATA 2-4 BMC_LAN USB01(3.0) VGA1 VGA USB01(3.0) BMC JPM1 JPM1 JPM1 JPM1 JPM1 JPM1 JPM1 JPM1 JPM1 JPM1 JPM1 JPM1 JPM1 JPM1 JPM1 JPM1 JPM1 JPM1 JPM1 JPM1 JPM1 JPM1 JPM1 JPM1 JPM1 JPM2 JPM2 JPM2 JPM2 JPM2 JPM2 JPM2 JPM2 JPM2 JPM2 JPM2 JPM2 JPM2 JPM2 JPM2 JPM2 JPM2 JPM2 JPM2 JPM2 JPM2 JPM2 JPM2 JPM2 JPM2 JPM3 JPM3 JPM3 JPM3 JPM3 JPM3 JPM3 JPM3 JPM3 JPM3 JPM3 JPM3 JPM3 JPM3 JPM3 JPM3 JPM3 JPM3 JPM3 JPM3 JPM3 JPM3 JPM3 JPM3 JPM3 JPM4 JPM4 JPM4 JPM4 JPM4 JPM4 JPM4 JPM4 JPM4 JPM4 JPM4 JPM4 JPM4 JPM4 JPM4 JPM4 JPM4 JPM4 JPM4 JPM4 JPM4 JPM4 JPM4 JPM4 JPM4 JPM5 JPM5 JPM5 JPM5 JPM5 JPM5 JPM5 JPM5 JPM5 JPM5 JPM5 JPM5 JPM5 JPM5 JPM5 JPM5 JPM5 JPM5 JPM5 JPM5 JPM5 JPM5 JPM5 JPM5 JPM6 JPM6 JPM6 JPM6 JPM6 JPM6 JPM6 JPM6 JPM6 JPM6 JPM6 JPM6 JPM6 JPM6 JPM6 JPM6 JPM6 JPM6 JPM6 JPM6 JPM6 JPM6 JPM7 JPM7 JPM7 JPM7 JPM7 JPM7 JPM7 JPM7 JPM7 JPM7 JPM7 JPM7 JPM7 JPM7 JPM7 JPM7 JPM7 JPM7 JPM7 JPM7 JPM7 JPM7 JPM8

Figure 7-3. Location of the MB Power LED

1. Make sure that no short circuits exist between the motherboard and chassis.
2. Disconnect all cables from the motherboard, including those for the keyboard and mouse.
3. Remove all add-on cards.
4. Install a CPU, a heatsink, connect the internal speaker (if applicable), and the power LED to the motherboard. Make sure that the heatsink is fully seated.
5. Use the correct type of onboard CMOS battery as recommended by the manufacturer. Check to verify that it still supplies \~3VDC. If it does not, replace it with a new one. Warning: To avoid possible explosion, do not install the battery upside down.
6. Verify that all jumpers are set to their default positions.
7. Check that the power supplies' input voltage operate at 100-120V or 180-240V.
8. Turn the power switch on and off to test the system

System Boot Failure

If the system does not display POST (Power-On-Self-Test) or does not respond after the power is turned on, check the following:

Turn on the system with only one DIMM module installed. If the system boots, check for bad DIMM modules or slots by following the Memory Errors Troubleshooting procedure below.

Memory Errors

1. Make sure that the DIMM modules are properly and fully installed.
2. Confirm that you are using the correct memory. Also, it is recommended that you use the same memory type and speed for all DIMMs in the system. See Section 3.4 for memory details.
3. Check for bad DIMM modules or slots by swapping modules between slots and noting the results.
4. Check the power supply voltage 115V/230V switch.

Losing the System's Setup Configuration

1. Make sure that you are using a high quality power supply. A poor quality power supply may cause the system to lose the CMOS setup information.

2. The battery on your motherboard may be old. Check to verify that it still supplies \~3VDC. If it does not, replace it with a new one.

3. If the above steps do not fix the setup configuration problem, contact your vendor for repairs.

When the System Becomes Unstable

If the system becomes unstable during or after OS installation, check the following:

1. CPU/BIOS support: Make sure that your CPU is supported and that you have the latest BIOS installed in your system.
2. Memory support: Make sure that the memory modules are supported by testing the modules using memtest86 or a similar utility.
   Note: Refer to the product page on our website at http://www.supermicro.com for memory and CPU support and updates.
3. HDD support: Make sure that all hard disk drives (HDDs) work properly. Replace the bad HDDs with good ones.
4. System cooling: Check the system cooling to make sure that all heatsink fans and CPU/system fans, etc., work properly. Check the hardware monitoring settings in the BMC to make sure that the CPU and system temperatures are within the normal range. Also check the front panel Overheat LED and make sure that it is not on.
5. Adequate power supply: Make sure that the power supply provides adequate power to the system. Make sure that all power connectors are connected. Please refer to our website for more information on the minimum power requirements.
6. Proper software support: Make sure that the correct drivers are used.

If the system becomes unstable before or during OS installation, check the following:

1. Source of installation: Make sure that the devices used for installation are working properly, including boot devices such as CD.
2. Cable connection: Check to make sure that all cables are connected and working properly.
3. Using the minimum configuration for troubleshooting: Remove all unnecessary components (starting with add-on cards first), and use the minimum configuration (but with a CPU and a memory module installed) to identify the trouble areas. Refer to the steps listed in Section 7.3 above for proper troubleshooting procedures.

4. Identifying bad components by isolating them: If necessary, remove a component in question from the chassis, and test it in isolation to make sure that it works properly. Replace a bad component with a good one.

5. Check and change one component at a time instead of changing several items at the same time. This will help isolate and identify the problem.

6. To find out if a component is good, swap this component with a new one to see if the system will work properly. If so, then the old component is bad. You can also install the component in question in another system. If the new system works, the component is good and the old system has problems.

7.4 Crash Dump Using BMC

In the event of a processor internal error (IERR) that crashes your system, you may want to provide information to support staff. You can download a crash dump of status information using BMC. The BMC manual is available at https://www.supermicro.com/en/solutions/management-software/bmc-resources.

Check BMC Error Log

1. Access the BMC web interface.

Dashboard System Component Info Health Event Log Configuration Remote Control Maintenance Advanced Settings Filter x Sensor-specific x Threshold x Genetic x OEM x Unspecified Health Event Log Clear Health Event Log Export to Excel Severity Date/Time Sensor Type Categories Description Event Type 2020-10-15 18:57:08 ACPowerOn [OEM] First AC Power on - Assection Sensor-specific 2020-10-15 18:56:43 System MC [OEM] Dedicated LAN Link Up - Assection Sensor-specific 2020-10-15 17:41:37 ACPowerOn [OEM] First AC Power on - Assection Sensor-specific 2020-10-15 17:41:17 System MC [OEM] Dedicated LAN Link Up - Assection Sensor-specific 2020-10-15 17:41:14 System MC [OEM] Dedicated LAN Link Down - Assection Sensor-specific 2020-10-15 17:41:12 System MC [OEM] Dedicated LAN Link Up - Assection Sensor-specific 1.6/4

Figure 7-4. BMC Event Log

1. Click the Server Health tab, then Event Log to verify an IERR error.

In the event of an IERR, the BMC executes a crash dump. You must download the crash dump and save it.

7.5 UEFI BIOS Recovery

Warning: Do not upgrade the BIOS unless your system has a BIOS-related issue. Flashing the wrong BIOS can cause irreparable damage to the system. In no event shall Supermicro be liable for direct, indirect, special, incidental, or consequential damages arising from a BIOS update. If you do update the BIOS, do not shut down or reset the system while the BIOS is updating to avoid possible boot failure.

Overview

The Unified Extensible Firmware Interface (UEFI) provides a software-based interface between the operating system and the platform firmware in the pre-boot environment. The UEFI specification supports an architecture-independent mechanism that will allow the UEFI OS loader stored in an add-on card to boot the system. The UEFI offers clean, hands-off management to a computer during system boot.

Recovering the UEFI BIOS Image

A UEFI BIOS flash chip consists of a recovery BIOS block and a main BIOS block (a main BIOS image). The recovery block contains critical BIOS codes, including memory detection and recovery codes for the user to flash a healthy BIOS image if the original main BIOS image is corrupted. When the system power is turned on, the recovery block codes execute first. Once this process is complete, the main BIOS code will continue with system initialization and the remaining POST (Power-On Self-Test) routines.

Note 1: Follow the BIOS recovery instructions below for BIOS recovery when the main BIOS block crashes.

Note 2: When the BIOS recovery block crashes, you will need to follow the procedures to make a Returned Merchandise Authorization (RMA) request. Also, you may use the Supermicro Update Manager (SUM) Out-of-Band (https://www.supermicro.com.tw/products/nfo/SMS_SUM.cfm) to reflash the BIOS.

Recovering the Main BIOS Block with a USB Device

This feature allows the user to recover the main BIOS image using a USB-attached device without additional utilities used. A USB flash device such as a USB Flash Drive, or a USB CD/DVD ROM/RW device can be used for this purpose. However, a USB Hard Disk drive cannot be used for BIOS recovery at this time.

The file system supported by the recovery block is FAT (including FAT12, FAT16, and FAT32) which is installed on a bootable or non-bootable USB-attached device. However, the BIOS might need several minutes to locate the SUPER.ROM file if the media size becomes too large due to the huge volumes of folders and files stored in the device.

To perform UEFI BIOS recovery using a USB-attached device, follow the instructions below.

1. Using a different machine, copy the "Super.ROM" binary image file into the Root "\" directory of a USB device or a writable CD/DVD.
   Note 1: If you cannot locate the "Super.ROM" file in your drive disk, visit our website at www.supermicro.com to download the BIOS package. Extract the BIOS binary image into a USB flash device and rename it "Super.ROM" for the BIOS recovery use.
   Note 2: Before recovering the main BIOS image, confirm that the "Super.ROM" binary image file you download is the same version or a close version meant for your motherboard.
2. Insert the USB device that contains the new BIOS image ("Super.ROM") into your USB drive and reset the system when the following screen appears.
3. After locating the healthy BIOS binary image, the system will enter the BIOS Recovery menu as shown below.

SUPERIOR BMC IP:10.132.161.13 PEI--Could Not Find Recovery Image... 87

Note: At this point, you may decide if you want to start the BIOS recovery. If you decide to proceed with BIOS recovery, follow the procedures below.

Option Setup Utility - Copyright (C) 2017 American Megatrends, Inc. Main Advanced Earnings PDF Recovery Security Boot Save & Exit Please select blocks you want to update Reset Value (Enabled) Boot Black Update (Enabled) ► Proceed with flash update Set this option to reset NORM to default values H: Select Screen T4: Select Item Enter Select +/- Change Out... F1: General Help F2: Previous Values F3: Optimized Defaults F4: Save & Exit ESC: EXIT Version 2.19.1960 Copyright 12/17 September 2017

1. When the screen as shown above displays, use the arrow keys to select the item "Proceed with flash update" and press the key. You will see the BIOS recovery progress as shown in the screen below.

Note: Do not interrupt the BIOS flashing process until it has completed.

1. After the BIOS recovery process is complete, press any key to reboot the system.

2. Using a different system, extract the BIOS package into a USB flash drive.

Auto Setup Utility - Copyright (C) 2017 American Newsrends, Inc. Recovery WARNING System firmware is being updated. Keyboard is locked. DO NOT TURN THE POWER OFF IT Price firmware update is completed press any key to reboot the system Program new data write new boot block... 17% +: Select Screen T4: Select item Enter: Select +/-: Change Opt. F1: General Help F2: Previous Values F3: For Indicnd Default F4: Save a Exit EKC: Exit Version 2.18.1266. Copyright (C) 2017 American Newsrends, Inc.

1. Press continuously during system boot to enter the BIOS Setup utility. From the top of the tool bar, select Boot to enter the submenu. From the submenu list, select Boot

Ratio Setup Utility - Copyright (CF 2017 American Regatrends, Inc. Recovery) HABNINIT System firmware is being updated. Keyboard is locked. DO NOT THAN THE POWER OFF !!! Once firmware update is completed press any key to reboot the system Flash update Flash update completed. Press any key to reset the system #+: Select Screen I1: Select Items Enter: Select +/-: Change Opt. F1: General Help F2: Previous Values F3: Optimized Defaults F4: Save & Exit SRS: Exit Version 2.15.1956. Copyright (CF 2017 American Repetrends, Inc.

Option #1 as shown below. Then, set Boot Option #1 to [UEFI AP:UEFI: Built-in EFI Shell]. Press to save the settings and exit the BIOS Setup utility.

1. When the UEFI Shell prompt appears, type fs# to change the device directory path. Go to the directory that contains the BIOS package you extracted earlier from Step 6. Enter flash.nsh BIOSname.### at the prompt to start the BIOS update process.

Auto Setup Utility - Copyright (C) 2017 American Megatrends, Inc. Main Advanced Event Log: IPRC Security Boot Save & Exit Boot Configuration Boot mode select [Dual] LEGACY to EFT support [Disabled] FIXED BOOT ORDER Priorities Boot Option #3 [IDEF1 HP/IDEF1: Max...] Boot Option #2 [COM/DVD] Boot Option #3 [USB Hard Disk] Boot Option #4 [USB CO/DVI] Boot Option #5 [USB Key/DimDisk] Boot Option #6 [USB Floopy] Boot Option #7 [USB Lan] Boot Option #8 [Network: IBM GE SI,...] Boot Option #9 [IDEF1 Hard Disk] Boot Option #10 [IDEF1 CO/DVI] Boot Option #11 [IDEF1 USB Hard Disk] Boot Option #12 [IDEF1 USB CO/DVI] Boot Option #13 [IDEF1 USB Key/IDEF1...] Boot Option #14 [IDEF1 USB Floopy] Boot Option #15 [IDEF1 USB Lan] Boot Option #16 [IDEF1 Network] Boot Option #17 [Hard Disk] Add New Boot Option Sets the system boot order +/- Select Screen T4: Select Item Enter: Select +/-: Change Opt, F1: Genera) Help F2: Precious Values F3: Optimized Defaults F4: Save & Exit ESC: Exit Version 2.19.126t Copyright (S) SRL American Megatrends, Inc.

Note: Do not interrupt this process until the BIOS flashing is complete.

LEFI Interactive Shell v2.1 DSK_II LEFI v2.50 (American Regatrends, 0x0089000) Mapping value F50: A130071;A001001;BLK1; F50:001000;PCL(0x14,0x0)/USS(0x1,0x0);H011,MER,HO791D72,0x896,0x1 DR3532 BLK9: A130071 F50:001000;PCL(0x14,0x0)/USS(0x1,0x0) Press ESR, No 1 seconds to include startup.msh or any other key to continue. Shell (FSI) F50: V 2E FROOG F50:VFW005V CB SAPME2.035201T F50:VFW005V SAPME2.035201T - F50H.msh X13P07.354

1. The screen above indicates that the BIOS update process is complete. When you see the screen above, unplug the AC power cable from the power supply, clear CMOS, and plug

Done. Access Once Port Ex +Reach Index #451: 0x18 Done. * Program BIOS and ME (including AFT) region... * * Ant Hardware Update Utility (AFT-01.13IT) Copyright ©2017 American Regetrands Doc. All Rights Reserved. OPUD = 50662 Reading Flash ....... done - ME Data Size checking .Ok - PBT checkSurs ....... Ok - Check RootLayout ....... Ok Erasing Root Block ....... done Updating Root Block ....... done Verifying Root Book ....... done Drilling Main Block ....... 0x0132899 COD

the AC power cable in the power supply again to power on the system.

1. Press continuously to enter the BIOS Setup utility.

Verifying NLL Book ....... done - Update success for ROR - Update success for SE... - Successful update Recovery Loader to (PRR) - Successful update NT5011- - Successful update FMR1- - Successful update RPA, 1081 and NT5011- - Successful Update FLOD and UTOI11- - ML entire image update success 11 WARNING : System must power-off to have the changes take effect! Moving FSN:VARU000S#KPM2E_03162057140x64_at1 -> FSN:VARU000S#KPM2E_00362017_v df, sac - [s] Moving FSN:VARU000S#KPM2E_0316205749aefu64_at1 -> FSN:VARU000S#KPM2E_0316201 Trafuel, sac - [s] ********************************************************************** * Please ignore this 'Shell cannot read from file - double error' * Learning message due to it does not impact flashing process. ********************************************************************** (* *) Sailing * Delete successful. FOS:>

1. Press to load the default settings.
2. After loading the default settings, press to save the settings and exit the BIOS Setup utility.

7.6 CMOS Clear

JBT1 is used to clear CMOS, which will also clear any passwords. Instead of pins, this jumper consists of contact pads to prevent accidentally clearing the contents of CMOS.

To Clear CMOS

1. First power down the system completely.
2. Remove the node from the chassis to access the motherboard.
3. Remove the onboard battery from the motherboard.
4. Short the CMOS pads with a metal object such as a small screwdriver for at least four seconds.
5. Remove the screwdriver or shorting device.
6. Re-install the battery.
7. Replace the cover, reconnect the power cords and power on the system.

Notes: Clearing CMOS will also clear all passwords.

Do not use the PW_ON connector to clear CMOS.

JBT1 contact pads

7.7 Where to Get Replacement Components

If you need replacement parts for your system, to ensure the highest level of professional service and technical support, purchase exclusively from our Supermicro Authorized Distributors/System Integrators/Resellers. A list can be found at: http://www.supermicro.com. Click the "Where to Buy" tab.

7.8 Reporting an Issue

Technical Support Procedures

Before contacting Technical Support, please take the following steps. If your system was purchased through a distributor or reseller, please contact them for troubleshooting services. They have the best knowledge of your specific system configuration.

1. Please review the Troubleshooting Procedures in this manual and Frequently Asked Questions on our website before contacting Technical Support.
2. BIOS upgrades can be downloaded from our website. Note: Not all BIOS can be flashed depending on the modifications to the boot block code.

3. If you still cannot resolve the problem, include the following information when contacting us for technical support:

4. System, motherboard, and chassis model numbers and PCB revision number

5. BIOS release date/version (this can be seen on the initial display when your system first boots up)
6. System configuration

An example of a Technical Support form is posted on our website. Distributors: For immediate assistance, please have your account number ready when contacting our technical support department by email.

Returning Merchandise for Service

A receipt or copy of your invoice marked with the date of purchase is required before any warranty service will be rendered. You can obtain service by calling your vendor for a Returned Merchandise Authorization (RMA) number. When returning to the manufacturer, the RMA number should be prominently displayed on the outside of the shipping carton, and mailed prepaid or hand-carried. Shipping and handling charges will be applied for all orders that must be mailed when service is complete.

For faster service, RMA authorizations may be requested online (http://www.supermicro.com/support/rma/).

Whenever possible, repack the chassis in the original Supermicro carton, using the original packaging material. If these are no longer available, be sure to pack the chassis securely, using packaging material to surround the chassis so that it does not shift within the carton and become damaged during shipping.

This warranty only covers normal consumer use and does not cover damages incurred in shipping or from failure due to the alteration, misuse, abuse or improper maintenance of products.

During the warranty period, contact your distributor first for any product problems.

Vendor Support Filing System

For issues related to Intel, use the Intel IPS filing system:

https://www.intel.com/content/www/us/en/design/support/ips/training/welcome.html

For issues related to Red Hat Enterprise Linux, since it is a subscription based OS, contact your account representative.

7.9 Feedback

Supermicro values your feedback as we strive to improve our customer experience in all facets of our business. Please email us at [e-mail] to provide feedback on our manuals.

Appendix A

Standardized Warning Statements for AC Systems

About Standardized Warning Statements

The following statements are industry standard warnings, provided to warn the user of situations which have the potential for bodily injury. Should you have questions or experience difficulty, contact Supermicro's Technical Support department for assistance. Only certified technicians should attempt to install or configure components.

Read this appendix in its entirety before installing or configuring components in the Supermicro chassis.

These warnings may also be found on our website at http://www.supermicro.com/about/policies/safety_information.cfm.

Warning Definition

Warning! This warning symbol means danger. You are in a situation that could cause bodily injury. Before you work on any equipment, be aware of the hazards involved with electrical circuitry and be familiar with standard practices for preventing accidents.

警告の定義

この警告サインは危険を意味します。

Installation Instructions

Warning! Read the installation instructions before connecting the system to the power source.

設置手順書

Warning! This product relies on the building's installation for short-circuit (overcurrent) protection. Ensure that the protective device is rated not greater than: 250 V, 20 A.

サーキット・ブレーカー

Power Disconnection Warning

Warning! The system must be disconnected from all sources of power and the power cord removed from the power supply module(s) before accessing the chassis interior to install or remove system components.

CAUTION: This unit has redundant power sources. Please disconnect all the power cords before servicing.

電源切断の警告

Equipment Installation

Warning! Only trained and qualified personnel should be allowed to install, replace, or service this equipment.

機器の設置

Warning! This unit is intended for installation in restricted access areas. A restricted access area can be accessed only through the use of a special tool, lock and key, or other means of security. (This warning does not apply to workstations).

アクセス制限区域

Warning! There is the danger of explosion if the battery is replaced incorrectly. Replace the battery only with the same or equivalent type recommended by the manufacturer. Dispose of used batteries according to the manufacturer's instructions

電池の取り扱い

Redundant Power Supplies

Warning! This unit might have more than one power supply connection. All connections must be removed to de-energize the unit.

冗長電源装置

Warning! Hazardous voltage or energy is present on the backplane when the system is operating. Use caution when servicing.

バックプレーンの電圧

Comply with Local and National Electrical Codes

Warning! Installation of the equipment must comply with local and national electrical codes.

地方および国の電気規格に準拠

Warning! Ultimate disposal of this product should be handled according to all national laws and regulations.

製品の廃棄

Warning! Hazardous moving parts. Keep away from moving fan blades. The fans might still be turning when you remove the fan assembly from the chassis. Keep fingers, screwdrivers, and other objects away from the openings in the fan assembly's housing.

ファン・ホットスワップの警告

Power Cable and AC Adapter

Warning! When installing the product, use the provided or designated connection cables, power cables and AC adaptors. Using any other cables and adaptors could cause a malfunction or a fire. Electrical Appliance and Material Safety Law prohibits the use of UL or CSA-certified cables (that have UL/CSA shown on the cord) for any other electrical devices than products designated by Supermicro only.

電源コードとACアダプター

Processors (per node)

Dual Intel Xeon Scalable Family 3rd Gen/4th Gen Series Processors (in Socket P+) with a thermal design power (TDP) of up to 250W and three UPIs (UltraPath Interconnect) of up to 11.2GT/s Note: Refer to the motherboard specifications pages on our website for updates to supported processors.

Chipset

Intel PCH C621A (LBG-R)

BIOS

AMI BIOS

ACPI 3.0 or later, PCI firmware 4.0 support, BIOS rescue hot-key, SPI dual/quad speed support, riser card auto detection support, RTC (Real Time Clock) wakeup, and SMBIOS 3.0 or later

Memory (per node)

16 DRAM DIMMs plus two PMems (Intel® Optane™ PMem 200 Series). Up to 4TB RDIMM, DDR4-3200MHz in 16 memory slots

Storage Drives

Front hot-swappable drives include:

12 NVMe/SATA drives for 620BT-DNTR

12 NVMe/SAS/SATA drives for 620BT-DNC8R

Internal M.2 drives per node include:

A default carrier card supporting up to two M.2 NVMe/SATA SSDs

An optional carrier card supporting up to two NVMe SSDs

Internal Connector:

VROC key header

PCI Expansion Slots (per node)

Two PCIe 4.0 x16 slots

Networking (per node)

One AIOM or any compliant OCP 3.0 SFF Network Interface Card

One dedicated LAN port for BMC

Input/Output (per node)

Two USB 3.0 ports

One VGA port

Motherboard (per node)

X12DPT-B6 (WxL) 7.4" x 18.86" (187.96 mm x 479.04 mm)

Chassis

CSE-827BD-R2K22P; 2U rackmount; (WxHxD) 17.6 x 3.5 x 30.5 in. (446 x 88 x 774 mm)

System Cooling

Four 8-cm mid-chassis fans, two CPU heat sinks per node, and one air shroud per node

Power Supply

Model: PWS-2K22A-1R, 2200 W redundant module, 80Plus Titanium level

AC Input Voltages: 100-240 VAC

Rated Input Current:

1800W: 200-220Vac

1980W: 220-230Vac

2090W: 230-240Vac

2090W: 180-220Vac (for UL/cUL only)

2200W: 220-240Vac (for UL/cUL only)

2090W: 230-240Vdc (for CCC only)>

Rated Input Frequency: 50-60 Hz

Rated Output Power: +12 V

Max: 100A / Min: 0A (100-127Vac)

Max: 150A / Min: 0A (200-220Vac)

Max: 165A / Min: 0A (220-230Vac)

Max: 174.17A / Min: 0A (230-240Vac)

Max: 174.17A / Min: 0A (180-220Vac, UL/cUL only)

Max: 183.33A / Min: 0A (220-240Vac, UL/cUL only)

Max: 174.17A / Min: 0A (230-240Vdc, CCC only)

Standby +12Vsb: Max: 2.1A / Min: 0A

Operating Environment

Operating Temperature: 10° to 35° C (50° to 95° F)

Non-operating Temperature: -40^ to 60^ C ( -40^ to 140^ F)

Operating Relative Humidity: 8% to 90% (non-condensing)

Non-operating Relative Humidity: 5% to 95% (non-condensing)

Regulatory Compliance

FCC, ICES, CE, VCCI, RCM, NRTL, CB

Applied Directives, Standards

EMC/EMI: 2014/30/EU (EMC Directive)

FCC Part 15

ICE-003

VCC 32-1

AS/NZS CISPR 32

EN55032

EN55035

CISPR 24

EN 61000-3-2

EN 61000-3-3

EN 61000-4-2

EN 61000-4-3

EN 61000-4-4

EN 61000-4-5

EN 61000-4-6

EN 61000-4-8

EN 61000-4-11

Green Environment:

2011/65/EU (RoHS Directive)

EC 1907/2006 (REACH)

2012/19/EU (WEEE Directive)

California Proposition 65

Product Safety: 2014/35/EU (LVD Directive)

UL/CSA 60950-1, 62368-1 (USA and Canada)

IEC/BS/EN 60950-1, 62368-1

Perchlorate Warning

California Best Management Practices Regulations for Perchlorate Materials: This Perchlorate warning applies only to products containing CR (Manganese Dioxide) Lithium coin cells. "Perchlorate Material-special handling may apply. See www.dtsc.ca.gov/hazardouswaste/perchlorate"

Appendix C

CPU-Based RAID for NVMe

Intel® Virtual RAID on CPU (Intel VROC) is an enterprise RAID solution for NVMe SSDs directly attached to Intel Xeon Scalable processors. Intel Volume Management Device (VMD) is an integrated controller inside the CPU PCIe root complex.

• A single processor supports up to 12 NVMe SSDs and up to 6 RAID arrays.
• A dual processor system supports up to 24 NVMe SSDs and 12 RAID arrays.

Strip sizes are 4K, 8K, 16K, 32K, 64K, 128K.

Requirements and Restrictions

• Intel VROC is only available when the system is configured for UEFI boot mode.
• To enable the mdadm command and support for RSTe, install the patch from

- Linux: https://downloadcenter.intel.com/download/28158/Intel-Virtual-RAID-on-CPU-Intel-VROC-and-Intel-Rapid-Storage-Technology-enterprise-Intel-RSTe-Driver-for-Linux-

- Windows: https://downloadcenter.intel.com/download/28108/Intel-Virtual-RAID-on-CPU-Intel-VROC-and-Intel-Rapid-Storage-Technology-enterprise-Intel-RSTe-Driver-for-Windows-

• To enable Intel VROC, a hardware key must be inserted on the motherboard, and the appropriate processor's Virtual Management Devices must be enabled in the BIOS setup.
• It is possible to enable Intel VROC without a hardware key installed, but only RAID0 will be enabled.
• Intel VROC is not compatible with secure boot. This feature must be disabled.
• When creating bootable OS RAID1 devices, you must have both devices on the same CPU, and a VMD on that CPU.
• Spanning drives when creating RAID devices is not recommended due to performance issues, even though it is supported.

Supported SSDs and Operating Systems

To see the latest support information: https://www.intel.com/content/www/us/en/support/articles/000030310/memory-and-storage/ssd-software.html

Additional Information

Additional information is available on the product page for the Supermicro add-on card and the linked manuals.

www.supermicro.com/products/accessories/addon/AOC-VROCxxxMOD.cfm

C.1 Hardware Key

The Intel VROC hardware key is a license key that detects the Intel VROC SKU and activates the function accordingly. The key must be plugged into the Supermicro motherboard (connector JRK1). The key options are:

Intel® VROC Keys
VROC Package Description Part Number Intel MM Number
Standard	RAID 0, 1, 10Supports 3rd party SSDs	AOC-VROCSTNMOD 95	1605
Premium	RAID 0, 1, 5, 10Supports 3rd party SSDs	AOC-VROCPREMOD 95	1606
Intel SSD only	RAID 0, 1, 5, 10Supports Intel SSDs only	AOC-VROCINTMOD 95	822

VROC_HW_KEY PIN 1 GND KEY1 GND KEY2

Figure C-1. Intel® VROC RAID Key and Motherboard Connector JRK1

C.2 Enabling NVMe RAID

RAID for NVMe SSDs must be enabled through the UEFI BIOS.

1. Install the patch as described in the Restrictions and Requirements section on a previous page.
2. Reboot the server.
3. Press [DEL] key to enter BIOS.
4. Switch to Advanced > Chipset Configuration > North Bridge > IIO Configuration > Intel VMD Technology > Intel VMD for Volume Management on CPU2.
5. Enable the VMD.

VMD Config for PStack0 Inteio VMD for Volume Management [Enable] Device for PStack0 CPU2 SXE2 NVMe/SRS VMD port 1B [Enable] CPU3 SXE2 NVMe/GAS VMD port 1B [Enable] Hot Plug Capable [Enable] VMD Config for PStack1 Inteio VMD for Volume Management [Disable] Device for PStack1 VMD Config for PStack2 Inteio VMD for Volume Management [Disable] Device for PStack2 Enable/DI Volume Max Technology ++: Select TI: Select Enter: Set +/-= Change E1: General

Figure C-2. BIOS VMD Settings

1. Press [F4] to save the configuration and reboot the system.
2. Press [DEL] to enter BIOS.
3. Switch to Advanced > Intel® Virtual RAID on CPU > All Intel VMD Controllers > Create RAID Volume.
4. Set Name.
5. Set RAID Level.

MDD Setup Utility - © Copyright ©100 AMPLEX MEAGRENDI. DC... Advanced Create MDD Volume Name: MDD Level: Make MDD scanned over VMS Controller [ ] Select Event: DETEI DESPECKSHWHTI GAVELFDSISAGOPAPPSI [6] DETEI DESPECKSHWHTI GAVELFDSISAGOPAPPSI [6] DETEI DESPECKSHWHTI GAVELFDSISAGOPAPPSI [6] DETEI DESPECKSHWHTI GAVELFDSISAGOPAPPSI [6] Strike Size: $20K30 Capacity: 8579.43 Create Volume E = To Select Disk << Select Servers B4: Select Znew Enter Select +/-: Change Net. B1: General Hio F2: Previous Values F3: Available Defaults F4: Down & Exit BDC: Exit

Figure C-3. Created Volume without enabling RAID spanned over VMD controller

Add to USM (File) - [SINFLIGHT 01: 228 AMERICAN RESOURCES INC. Advanced Create AMD Volume Name: HAMD Level: Single AMD approved over AMD Controller X = For Select Disk Select Disk: INTL, SISPEKONOMITT SHAPLETOSOOPHAPID INTL, SISPEKONOMITT SHAPLETOSOOPHAPID INTL, SISPEKONOMITT SHAPLETOSOOPHAPID IPCL, SISPEKONOMITT SHAPLETOSOOPHAPID Strip Size: Capacity (Watts) 1200KB 14000 KB + Create Volume = Select Windows Pkt Select Item Other: Select Any Change Not Pkt General Rate Pkt Previous Values Pkt Selected Defaults Pkt Save & Fail CNCI Exit

Figure C-4. Created Volume with enabling RAID spanned over VMD controller

1. If cross-controller RAID is required, select Enable RAID spanned over VMD Controller as shown in Figure C-4.
2. Select specific disks for RAID with an [X].

• RAID0: Select at least two [2 - 24] disks
• RAID1: Select only two disks
• RAID5: Select at least three [3 - 24] disks
• RAID10: Select only four disks

1. Select Strip Size (Default 64KB).
2. Select Create Volume.
3. If another RAID is needed, start again at step 6.
4. Press [F4] to save and reboot.

C.3 Status Indications

An LED indicator on the drive carrier shows the RAID status of the drive.

Drive Carrier Status LED Indicator
Status State (red)
Normal function Off
Locating 4 Hz blink
Fault Solid on
Rebuilding 1 Hz Blink

IBPI SFF 8489 Defined Status LED States

C.4 Hot-Swap Drives

Intel VMD enables hot-plug and hot-unplug for NVMe SSDs, whether from Intel or other manufacturers. Under vSphere ESXi, several steps are necessary to avoid potential stability issues. See the information at link [1] below.

Hot-unplug

1. Prevent devices from being re-detected during rescan:

esxcli storage core claiming autoclaim --enabled=false

1. Unmount the VMFS volumes on the device. Check [2] for details.
2. Detach the device. Check [3] for details.
3. Physically remove the device.

Hot-plug

• Physically install the device.

ESXi will automatically discover NVMe SSDs, but a manual scan may be required in some cases.

Related Information Links

[1] https://kb.vmware.com/s/article/2151404
[2] https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.storage.doc/GUID-1B56EF97-F60E-4F21-82A7-8F2A7294604D.html
[3] https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.storage.doc/GUID-F2E75F67-740B-4406-9F0C-A2D99A698F2A.html