SuperServer SSG-640SP-DE1CR60 - Server Supermicro - Free user manual and instructions

USER MANUAL SuperServer SSG-640SP-DE1CR60 Supermicro

Front view of a server rack unit with hexagonal ventilation grilles and a 'SUPERMICK' logo on the front panel (no readable text beyond branding)

USER'S MANUAL

Revision 1.0a

The information in this User's Manual has been carefully reviewed and is believed to be accurate. The vendor assumes no responsibility for any inaccuracies that may be contained in this document, and makes no commitment to update or to keep current the information in this manual, or to notify any person or organization of the updates. Please Note: For the most up-to-date version of this manual, please see our website at www.supermicro.com.

Super Micro Computer, Inc. ("Supermicro") reserves the right to make changes to the product described in this manual at any time and without notice. This product, including software and documentation, is the property of Supermicro and/or its licensors, and is supplied only under a license. Any use or reproduction of this product is not allowed, except as expressly permitted by the terms of said license.

IN NO EVENT WILL Super Micro Computer, Inc. BE LIABLE FOR DIRECT, INDIRECT, SPECIAL, INCIDENTAL, SPECULATIVE OR CONSEQUENTIAL DAMAGES ARISING FROM THE USE OR INABILITY TO USE THIS PRODUCT OR DOCUMENTATION, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. IN PARTICULAR, SUPER MICRO COMPUTER, INC. SHALL NOT HAVE LIABILITY FOR ANY HARDWARE, SOFTWARE, OR DATA STORED OR USED WITH THE PRODUCT, INCLUDING THE COSTS OF REPAIRING, REPLACING, INTEGRATING, INSTALLING OR RECOVERING SUCH HARDWARE, SOFTWARE, OR DATA.

Any disputes arising between manufacturer and customer shall be governed by the laws of Santa Clara County in the State of California, USA. The State of California, County of Santa Clara shall be the exclusive venue for the resolution of any such disputes. Supermicro's total liability for all claims will not exceed the price paid for the hardware product.

FCC Statement: This equipment has been tested and found to comply with the limits for a Class A or Class B digital device pursuant to Part 15 of the FCC Rules. These limits are designed to provide reasonable protection against harmful interference when the equipment is operated in industrial environment for Class A device or in residential environment for Class B device. This equipment generates, uses, and can radiate radio frequency energy and, if not installed and used in accordance with the manufacturer's instruction manual, may cause harmful interference with radio communications. Operation of this equipment in a residential area is likely to cause harmful interference, in which case you will be required to correct the interference at your own expense.

California Best Management Practices Regulations for Perchlorate Materials: This Perchlorate warning applies only to products containing CR (Manganese Dioxide) Lithium coin cells. "Perchlorate Material-special handling may apply. See www.dtsc.ca.gov/hazardouswaste/perchlorate".

WARNING: This product can expose you to chemicals including lead, known to the State of California to cause cancer and birth defects or other reproductive harm. For more information, go to www.P65Warnings.ca.gov.

The products sold by Supermicro are not intended for and will not be used in life support systems, medical equipment, nuclear facilities or systems, aircraft, aircraft devices, aircraft/emergency communication devices or other critical systems whose failure to perform be reasonably expected to result in significant injury or loss of life or catastrophic property damage. Accordingly, Supermicro disclaims any and all liability, and should buyer use or sell such products for use in such ultra-hazardous applications, it does so entirely at its own risk. Furthermore, buyer agrees to fully indemnify, defend and hold Supermicro harmless for and against any and all claims, demands, actions, litigation, and proceedings of any kind arising out of or related to such ultra-hazardous use or sale.

Manual Revision 1.0a

Release Date: January 24, 2022

VC

Unless you request and receive written permission from Super Micro Computer, Inc., you may not copy any part of this document. Information in this document is subject to change without notice. Other products and companies referred to herein are trademarks or registered trademarks of their respective companies or mark holders.

Copyright © 2022 by Super Micro Computer, Inc.

All rights reserved.

Printed in the United States of America

Preface

About this Manual

This manual is written for professional system integrators and PC technicians. It provides information for the installation and use of the server. Installation and maintenance should be performed by experienced technicians only.

Please refer to the SSG-640SP-DE1CR60 server specifications page on our website for updates on supported memory, processors and operating systems (http://www.supermicro.com).

Notes

For your system to work properly, please follow the links below to download all necessary drivers/utilities and the user's manual for your server.

• Supermicro product manuals: http://www.supermicro.com/support/manuals/
- Product drivers and utilities: https://www.supermicro.com/wdl/driver
- Product safety info: http://www.supermicro.com/about/policies/safety_information.cfm

If you have any questions, please contact our support team at:

[e-mail]

This manual may be periodically updated without notice. Please check the Supermicro website for possible updates to the manual revision level.

Secure Data Deletion

A secure data deletion tool designed to fully erase all data from storage devices can be found on our website: https://www.supermicro.com/about/policies/disclaimer.cfm?url=/wdl/utility/Lot9_Secure_Data_Deletion_Utility/

Warnings

Special attention should be given to the following symbols used in this manual.

Warning! Indicates important information given to prevent equipment/property damage or personal injury.

Warning! Indicates high voltage may be encountered when performing a procedure.

Contents

Contacting Supermicro....8

Chapter 1 Introduction

1.1 Overview....9
1.2 System Features ....10

Front View....10

Drive Carrier Indicators....11
Control Panel....12
Rear View....13
Accessing the Drive Bays....15

1.3 System Architecture ....18

Main Components....18
System Block Diagram....19
Drawer Architectural Diagram....20

1.4 Motherboard Layout....21

Quick Reference Table....22

Chapter 2 Server Installation

2.1 Overview....23
2.2 Unpacking the System 23
2.3 Preparing for Setup....23

Choosing a Setup Location....23
Rack Precautions....24
Server Precautions....24
Rack Mounting Considerations....24
Ambient Operating Temperature....24
Airflow 25
Mechanical Loading....25
Circuit Overloading 25
Reliable Ground....25

2.4 Installing the Rails....26

Installing the Rails onto the Rack....26

2.5 Installing the Chassis into the Rack....30

Loading HDDs into the Chassis on a Rack....32

Chapter 3 Maintenance and Component Installation

3.1 Removing Power ....34
3.2 Accessing the System....35

HDD Drawer Access 35

Opening the HDD Drawer ....35

3.3 Static-Sensitive Devices....36

Precautions ....36

3.4 Processor and Heatsink Installation....37

The Processor Carrier Assembly ....38

The Processor Heatsink Module (PHM)....40

Preparing the CPU Socket for Installation....41

Installing the PHM into the CPU Socket....42

Removing the PHM from the CPU Socket ....44

Removing the Processor Carrier Assembly from the PHM 45

Removing the Processor from the Processor Carrier Assembly 46

3.5 Memory....47

Memory Support....47

General Memory Population Requirements....47

PMem 200 Series Population table for X12DP Motherboards (w/16 Slots)....48

DIMM Installation ....50

DIMM Removal 50

3.6 Mezzanine Card Installation....51

AOM-SADPT-S 51

AOM-S3616....51

Installing the Mezzanine Card....52

3.7 Motherboard Battery....53

3.8 Storage Drives ....54

Hard Drive Carrier Indicators....54

Installing Hot-Swap 3.5" Hard Drives....55

Hard Drive Installation....56

Expander Module Replacement....57

3.9 System Cooling....58

Fans ....58

Checking the Server Air Flow....60

Overheating....60

3.10 Power Supply 61

Power Supply LEDs....61

3.11PCI Expansion Cards ....62

3.12 BMC 63

Chapter 4 Motherboard Connections

4.1 Power Connections ....64

4.2 Headers and Connectors ....65

4.3 Input/Output Ports 67

Rear I/O Ports....67

4.4 Jumpers....69

4.5 LED Indicators....71

4.6 Storage Ports 73

Chapter 5 Software

5.1 Microsoft Windows OS Installation....74

5.2 Driver Installation....76

5.3 SuperDoctor® 5....77

5.4 BMC....78

BMC ADMIN User Password 78

Chapter 6 Optional Components

6.1 Optional Parts List....79

6.2 Intel Virtual RAID on CPU (VROC)....80

Requirements and Restrictions....80

Supported SSDs and Operating Systems 80

Additional Information 81

Hardware Key 81

Enabling NVMe RAID....82

Status Indications....85

Hot Swap Drives 85

Hot-unplug 85

Hot-plug 85

Chapter 7 Troubleshooting and Support

7.1 Information Resources....86

Website 86

Direct Links for the SSG-640SP-DE1CR60 System 86

Direct Links for General Support and Information 86

7.2 Baseboard Management Controller (BMC)....87

7.3 Troubleshooting Procedures 88

No Power 88

No Video ....89

System Boot Failure ....89

Memory Errors 89

Losing the System's Setup Configuration....89

When the System Becomes Unstable....89

7.4 BIOS Error Beep (POST) Codes 91

Additional BIOS POST Codes 91

7.5 Crash Dump Using BMC....92

7.6 UEFI BIOS Recovery ....93

Overview 93

Recovering the UEFI BIOS Image....93

Recovering the Main BIOS Block with a USB Device....93

7.7 CMOS Clear....98

7.8 Where to Get Replacement Components....99

7.9 Reporting an Issue....99

Technical Support Procedures 99

Returning Merchandise for Service....99

Vendor Support Filing System 100

7.10 Feedback....100

7.11 Contacting Supermicro....101

Appendix A Standardized Warning Statements for AC Systems Appendix B System Specifications

Contacting Supermicro

Headquarters

Address: Super Micro Computer, Inc.

980 Rock Ave.

San Jose, CA 95131 U.S.A.

Tel: [phone]

Fax: [phone]

Email: [e-mail] (General Information)

[e-mail] (Technical Support)

Website: www.supermicro.com

Europe

Address: Super Micro Computer B.V.

's-Hertogenbosch, The Netherlands

Tel: [phone]

Fax: [phone]

Email: [e-mail] (General Information)

[e-mail] (Technical Support)

[e-mail] (Customer Support)

Website: www.supermicro.nl

Asia-Pacific

Address: Super Micro Computer, Inc.

3F, No. 150, Jian 1st Rd.

Zhonghe Dist., New Taipei City 235

Taiwan (R.O.C)

Tel: [phone]

Fax: [phone]

Email: [e-mail]

Website: www.supermicro.com.tw

Chapter 1

Introduction

1.1 Overview

This chapter provides a brief outline of the functions and features of the SuperStorage Server SSG-640SP-DE1CR60, which is based on the X12DSC-6 motherboard and the CSV-947STS-R2K63P chassis.

The following provides an overview of the specifications and capabilities.

System Overview
Motherboard	X12DSC-6
Chassis	CSV-947STS-R2K63P
Processor Support*	Dual 3rd Generation Intel® Xeon® Scalable processors
Memory*	4TB of 3DS LRDIMM/LRDIMM/3DS RDIMM/RDIMM DDR4 (288-pin) ECC memory with speeds of 3200/2933/2666 MHz in 16 memory slots and up to 4TB of Intel Optane PMem 200 Series with speeds of up to 3200 MHz
Drive Support*	60 x3.5/2.5" hot-swap SAS3/SATA3 drive bays (each node controls 30 drives)Two 7mm or one 15mm internal NVMe drive bays (optional)Two internal 2.5" SATA3 drive bays (per node)Two NVMe M.2 (per node)
Expansion Slots*	Three PCIe 4.0 x16 low-profile slots (CPU SLOT 1, 2, 3)Two PCIe 4.0 x2 NVMe M.2 slots (supports M-Key 2280 and 22110, on mezzanine board)
I/O Ports*	Two 10GBaseT Ethernet LAN portsOne RJ45 dedicated BMC LAN portOne serial (COM) portOne VGA portTwo USB 3.0 ports (on the rear I/O panel)
System Cooling*	Six 8-cm heavy duty fans
Power	Two redundant power supply modules2600W (Titanium Level)
Form Factor	4U 7 x 17.6 x 34.1 in. / 178 x 447 x 866mm (H x W x D)

Notes: A Quick Reference Guide can be found on the product page of the Supermicro website.

The following safety models associated with the SSG-640SP-DE1CR60 have been certified as compliant with UL or CSA: 947S60, 947S-26, 947S-R26X12.

1.2 System Features

The following views of the system display the main features. Refer to Appendix B for additional specifications.

Front View

Drawer Latches Control Panels SUPERMICR Rack Locks Drawer Locks

Figure 1-1. Front View

System Features: Front (per Node)
Feature Description
Control Panels	Two front control panels - top node and bottom node (see Control Panel section for details)
Drawer Latch Latch	to remove the drawer from the system
Drawer Lock Locks	the drawer into the chassis
Rack Lock Locks	the system into a rack
Hot-swap Drive Bays	60 *3.5" SAS3/SATA3 hot-swap drive bays

Drive Carrier Indicators

Each drive carrier has two LED indicators: an activity indicator and a status indicator. For RAID configurations using a controller, the meaning of the status indicator is described in the table below. For OS RAID or non-RAID configurations, some LED indications are not supported, such as hot spare. For VROC configurations, refer to the VROC appendix in this manual.

Hard Drive Carrier LED Indicators
	Color Blinking Pattern Behavior for Device
Activity LED	Blue Solid on Indicates a SAS drive
	Blue Off Indicates a SATA drive
	Blue Blinking Drive is actively being accessed
Status LED	Red Solid on Drive failed
	Red Blinking at 1Hz RAID is rebuilding
	Red Blinking at 3Hz Indicates a hot spare
	Red Blinking at 4Hz Locates a drive
	Red Off Idle

Control Panel

POWER Power NIC LED i Information LED UID Button

Figure 1-2. Control Panel (per Node)

Control Panel Features (per Node)
Feature Description
Power button	The main power button is used to apply or remove power from the power supply to the server. Turning off system power with this button removes the main power but maintains standby power. To perform many maintenance tasks, you must also unplug system before servicing.
NIC LED Indicates network activity on LAN port when flashing.
Information LED Alerts operator to several states, as noted in the table below.
UID Button/LED	The Unit ID (UID) button is used to turn on or off the blue UID LED to easily locate the server in racks and server banks. Solid blue indicates the UID has been activated locally to locate the server in a rack environment. Blinking blue indicates the UID has been activated using BMC to locate the server in a rack environment.

Information LED
Status Description
Continuously on and red	1. An overheat condition has occurred.2. An HDD(s) status has changed. (For details, please refer to theHard Drive Carrier Indicators section).
Blinking red (1Hz) Fan failure, check for an inoperative fan.
Blinking red (0.25Hz)	Power failure, check for a non-operational power supply.

Rear View

Power Supplies System Fans BMC LAN Port USB Ports COM Port VGA Port LAN Ports CPU1 CPU2

Figure 1-3. Rear View

System Features: Rear (Node A)

Feature Description

Power Supplies Two 2600W* redundant power supplies

System Fans Six 8cm counter-rotating fans

BMC LAN Port One dedicated BMC LAN port per node

COM Port One COM port per node

USB Ports Two USB 3.0 ports per node

VGA Port One VGA port per node

LAN Ports Two RJ45 LAN ports per node

*The 2600W power supply uses a C19 type power plug.
Note: The bottom tier is Node A and the top tier is Node B.

Expansion Card Chassis Slots (per Node)

Item Description

1 PCIe 4.0x16 low-profile slot

2 PCIe 4.0x16 low-profile slot

3 PCIe 4.0x16 low-profile slot

Notes: HHHL = half height, half length.
Full height = 4.2", low profile = 2.5", full length = 10.5", half length = 6.6"

Power Supply Indicators
Power Supply Condition Green LED	Amber LED
No AC Power to Power Supply OFF OFF
Power Supply critical events causing a shutdown/ failure/ OCP/ OVP/ Fan Fail/ OTP/ UVP	OFF Amber LED
Power Supply Warning Events Where the power supply continues to operate; High temperature; Over voltage; under voltage, etc	OFF 1Hz Blink Amber
AC present only 12vsb on (PS off) 1Hz Blink	Green OFF
Output ON and OK Green OFF
AC cord unplugged and in redundant mode	OFF Amber

Accessing the Drive Bays

Unlock and pull out the drawer to access the drive bays. The first two rows of drive bays can be accessed by sliding the drawer out to the first stop position. Each row includes a mid-row gap, two of which have expander modules that connect the drives to the motherboards.

Front view of a server rack unit with hexagonal heat exchanger and labeled 'SUPERMICRO' (no readable text beyond label)

Figure 1-4. Drawer View

The Storage Server SSG-640SP-DE1CR60 contains two field-serviceable expander modules, which are installed in the second and fourth drive bay rows. The first and third drive bay rows have their mid-row gap filled by non-functional fillers (dummy modules).

Dummy Modules Expander Modules

Figure 1-5. Location of Expander Modules

Looking down on a pulled-out drawer, the drive bays in the first row from left to right are identified as 0 to 14. Drive bays 15 to 29 comprise the second row. An expander module in the second row connects drive bays 0 to 29 to the server node physically positioned at the bottom (see figure below).

Drive bays 30 to 44 comprise the third row, while drive bays 45 to 59 comprise the fourth row. Another expander module in the fourth row connects drive bays 30 to 59 to the server node physically positioned at the top (see figure below).

Top Node Bottom Node Drives 45-59 Drives 30-44 Drives 15-29 Drives 0-14 Front of Chassis Top Node Drives Bottom Node Drives

Figure 1-6. Logical Drive Locations

1.3 System Architecture

This section covers the locations of the system electrical components, a system block diagram, and a motherboard layout with the connectors and jumpers called out.

Main Components

AOM Controller Internal tray 2x SuperCap mounting brackets 2x M.2 slots

Figure 1-7. Main Component Locations

System Features: Top
Feature Description
AOM Controller Add-on module
Internal Tray Storage	tray for two 2.5" slim SATA SSDs or two Supercaps (optional)
Supercap Mounting Brackets	SuperCap TFM module (optional)
M.2 Slots Two M.2 PCIe	4.0 NVMe slots

System Block Diagram

The block diagram below shows the connections and relationships between the subsystems and major components of the overall system.

X12DSC-6

graph TD
    subgraph Power Components
        CPU1["CPU1-B1"] --> DDR4["DDR4 2666/2933/200"]
        CPU1 --> DDR4_1["DDR4 2666/2933/200"]
        CPU1 --> DDR4_2["DDR4 2666/2933/200"]
        CPU1 --> DDR4_3["DDR4 2666/2933/200"]
        CPU1 --> DDR4_4["DDR4 2666/2933/200"]
        CPU1 --> DDR4_5["DDR4 2666/2933/200"]
        CPU1 --> DDR4_6["DDR4 2666/2933/200"]
        CPU1 --> DDR4_7["DDR4 2666/2933/200"]
        CPU1 --> DDR4_8["DDR4 2666/2933/200"]
        CPU1 --> DDR4_9["DDR4 2666/2933/200"]
        CPU1 --> DDR4_10["DDR4 2666/2933/200"]
        CPU1 --> DDR4_11["DDR4 2666/2933/200"]
        CPU1 --> DDR4_12["DDR4 2666/2933/200"]
        CPU1 --> DDR4_13["DDR4 2666/2933/200"]
        CPU1 --> DDR4_14["DDR4 2666/2933/200"]
        CPU1 --> DDR4_15["DDR4 2666/2933/200"]
        CPU1 --> DDR4_16["DDR4 2666/2933/200"]
        CPU1 --> DDR4_17["DDR4 2666/2933/200"]
        CPU1 --> DDR4_18["DDR4 2666/2933/200"]
        CPU1 --> DDR4_19["DDR4 2666/2933/200"]
        CPU1 --> DDR4_20["DDR4 2666/2933/200"]
        CPU1 --> DDR4_21["DDR4 2666/2933/200"]
        CPU1 --> DDR4_22["DDR4 2666/2933/200"]
        CPU1 --> DDR4_23["DDR4 2666/2933/200"]
        CPU1 --> DDR4_24["DDR4 2666/2933/200"]
        CPU1 --> DDR4_25["DDR4 2666/2933/200"]
        CPU1 --> DDR4_26["DDR4 2666/2933/200"]
        CPU1 --> DDR4_27["DDR4 2666/2933/200"]
        CPU1 --> DDR4_28["DDR4 2666/2933/200"]
        CPU1 --> DDR4_29["DDR4 2666/2933/200"]
        CPU1 --> DDR4_30["DDR4 2666/2933/200"]
        CPU1 --> DDR4_31["DDR4 2666/2933/200"]
        CPU1 --> DDR4_32["DDR4 2666/2933/200"]
        CPU1 --> DDR4_33["DDR4 2666/2933/200"]
        CPU1 --> DDR4_34["DDR4 2666/2933/200"]
        CPU1 --> DDR4_35["DDR4 2666/2933/200"]
        CPU1 --> DDR4_36["DDR4 2666/2933/200"]
        CPU1 --> DDR4_37["DDR4 2666/2933/200"]
        CPU1 --> DDR4_38["DDR4 2666/2933/200"]
        CPU1 --> DDR4_39["DDR4 2666/2933/200"]
        CPU1 --> DDR4_40["DDR4 2666/2933/200"]
        CPU1 --> DDR4_41["DDR4 2666/2933/200"]
        CPU1 --> DDR4_42["DDR4 2666/2933/200"]
        CPU1 --> DDR4_43["DDR4 2666/2933/200"]
        CPU1 --> DDR4_44["DDR4 2666/2933/200"]
        CPU1 --> DDR4_45["DDR4 2666/2933/200"]
        CPU1 --> DDR4_46["DDR4 2666/2933/200"]
        CPU1 --> DDR4_47["DDR4 2666/2933/200"]
        CPU1 --> DDR4_48["DDR4 2666/2933/200"]
        CPU1 --> DDR4_49["DDR4 2666/2933/200"]
        CPU1 --> DDR4_50["DDR4 2666/2933/200"]
        CPU1 --> DDR4_51["DDR4 2666/2933/200"]
        CPU1 --> DDR4_52["DDR4 2666/2933/200"]
        CPU1 --> DDR4_53["DDR4 2666/2933/200"]
        CPU1 --> DDR4_54["DDR4 2666/2933/200"]
        CPU1 --> DDR4_55["DDR4 2666/2933/200"]
        CPU1 --> DDR4_56["DDR4 2666/2933/200"]
        CPU1 --> DDR4_57["DDR4 2666/2933/200"]
        CPU1 --> DDR4_58["DDR4 2666/2933/200"]
        CPU1 --> DDR4_59["DDR4 2666/2933/200"]
        CPU1 --> DDR4_60["DDR4 2666/2933/200"]
        CPU1 --> DDR4_61["DDR4 2666/2933/200"]
        CPU1 --> DDR4_62["DDR4 2666/2933/200"]
        CPU1 --> DDR4_63["DDR4 2666/2933/200"]
        CPU1 --> DDR4_64["DDR4 2666/2933/200"]
        CPU1 --> DDR4_65["DDR4 2666/2933/200"]
        CPU1 --> DDR4_66["DDR4 2666/2933/200"]
        CPU1 --> DDR4_<ecel><nl>

Figure 1-8. System Block Diagram

Drawer Architectural Diagram

The Storage Server SSG-640SP-DE1CR60 contains four Interposer positions for expander modules, labelled A, B, C, and D. This interposer design allows each row of 15 drive bays to be connected via foldable SAS cables to a midplane attached to both motherboards. The bottom motherboard is connected to its dedicated expander module installed in position B. The top motherboard is connected to another dedicated expander module installed in position D.

Passive Expander Interposer A B C D Hot-serviceable Expanders Passive Mid-plane X12DSC-6 Motherboards Passive SAS3 Device Backplanes (15 devices each) SAS3 Cables

Figure 1-8. Drawer Architectural Diagram

1.4 Motherboard Layout

Below is a layout of the X12DSC-6 motherboard with jumper, connector and LED locations shown. See the table on the following page for descriptions. For detailed descriptions, pinout information and jumper settings, refer to Chapter 4 or the Motherboard Manual.

graph TD
    subgraph Computer Architecture
        A["JPL1"] --> B["LE4"]
        C["JUIDB1"] --> D["JLAN"]
        E["JJSBRJ45"] --> F["JUSBRJ45"]
        G["JUNDA"] --> H["JUNDA"]
        I["JUDI"] --> J["JUDI"]
        K["JWD1"] --> L["JWD1"]
        M["JRK1"] --> N["JRK1"]
        O["JCOM"] --> P["JCOM"]
        Q["LEDM1"] --> R["JLEDM1"]
        S["JPRG1"] --> T["JPRG1"]
        U["JPME2"] --> V["JPME2"]
        W["JBT1"] --> X["JBT1"]
        Y["BT1"] --> Z["BT1"]
        AA["LE3"] --> AB["LE3"]
        AC["JL1"] --> AD["S-SATA4"]
        AE["JVRM"] --> AF["JVRM"]
        AG["S-SATA5"] --> AH["JPM1"]
        AI["BCN1"] --> AJ["BCN1"]
        AK["JMCN3"] --> AL["JMCN3"]
        AM["JNVME1"] --> AN["JNVME1"]
        AO["JNVME2"] --> AP["JNVME2"]
        AQ["JMCN2"] --> AR["JMCN2"]
        AS["BCN2"] --> AT["BCN2"]
        AU["JMCN1"] --> AV["JMCN1"]
        AW["JPWR_NVME"] --> AX["JPWR_NVME"]
    end

    subgraph Hardware Architecture
        A
        C
        E
        G
        I
        O
        S
        AC
        AC
    end

    subgraph Control Structure
        A
        B
        D
        F
        H
        I
        P
        R
        S
        T
        X
        Y
        Z
    end

    subgraph Control Components
        A
        B
        D
        F
        H
        I
        P
        Q
    end

    subgraph Control Systems
        A
        B
        D
        F
        H
        I
        P
        Q
    end

    subgraph Control Modules
        A
        B
        D
        F
        H
        I
    end

    subgraph Control Components
        A
        B
        D
        F
        H
        I
    end

    subgraph Control Systems
        A
        B
        D
        F
        H
    end

    subgraph Control Structures
        A
        B
        D
        F
        H
    end

    subgraph Control Systems
        A
        B
        D
        F
    end

    subgraph Control Modules
        A
        B
        D
    end

    subgraph Control Systems
        A
        B
        D
    end

    subgraph Control Structures
        A
        B
        D
    end

    subgraph Control Systems
        A
        B
        D
    end

    subgraph Control Modules
        A
        B
        D
    end

    subgraph Control Systems
        A
        B
        D
    end

    subgraph Control Structures
        A
        B
        D
    end

    subgraph Control Systems
        A
        B
        D
    end

    subgraph Control Structures
        A
        B
        D
    end

    subgraph Control Systems
        A
        B
        D
    end

    subgraph Control Structures
        A
        B
        D
    end

    subgraph Control Systems
        A
        B
       D["X12DSC-8"]
    end

    subgraph Control Structures
            direction TB
            direction LR

Figure 1-9. Motherboard Layout

Notes:

- "■" indicates the location of pin 1.

- Jumpers/LED indicators not indicated are used for testing only and are not covered in this manual.

Quick Reference Table

Jumper Description Default Setting

JBT1 CMOS Clear Open (Normal)
JPL1 LAN1 Enable/Disable Pins 1-2 (Enabled)
JPME2 Manufacturing Mode Select Pins 1-2 (Normal)
JWD1 Watch Dog Timer Enable Pins 1-2 (Enabled, Reset)
JVRM VRMI	^2C Closed

Connector Description

BCN1, BCN2 Backplane connectors (connects to backplane BPN-SAS3-947SB)
JCOM1 Serial Port header
JFAN, JFANPW, FAN1, FAN2	System Cooling Fan headers
S-SATA4, S-SATA5	SATA 3.0 connection headers supported by the PCH
BMC_LAN	Dedicated BMC_LAN port
JMCN1, JMCN2, JMCN3	Mezzanine board docking connectors
JNVME1/JNVME2	PCIe 4.0 x4x4 NVMe slots (JNVME1 is supported by CPU1, JNVME2 is supported by CPU2)
JL1	Chassis Intrusion header (Note: Please connect a cable from the Chassis Intrusion header at JL1 to the chassis to receive an alert via BMC.)
JLAN	Back panel LAN ports (LAN1/LAN2)
VROC (JRK1)	Intel VROC RAID key for NVMe SSD
JTPM1	Trusted Platform Module (TPM)/Port 80 connector
JPCIE1 PCIe 3.0 x8 slot supported by CPU1
JPCIE2 PCIe 3.0 x16 slot supported by CPU1
JPCIE3 PCIe 3.0 x16 slot supported by CPU2
JPWR_NVME	Power connector for NVMe backplane devices
JUIDB1	Unit Identifier (UID) Switch
JUSBRJ45	Back panel USB 3.0 ports (USB0/1)
JVGA	VGA port header for front access

LED Description
Status

LEDM1	BMC Heartbeat LED	Blinking Green: BMC normal
LE3	Onboard Power LED	On: Onboard power on
LE4	UID (Unit Identifier) LED	Solid Blue: Unit identified

Chapter 2

Server Installation

2.1 Overview

This chapter provides advice and instructions for mounting your system in a server rack. If your system is not already fully integrated with processors, system memory etc., refer to Chapter 3 for details on installing those specific components.

Caution: Electrostatic Discharge (ESD) can damage electronic components. To prevent such damage to PCBs (printed circuit boards), it is important to use a grounded wrist strap, handle all PCBs by their edges and keep them in anti-static bags when not in use.

2.2 Unpacking the System

Inspect the box in which the SSG-640SP-DE1CR60 was shipped, and note if it was damaged in any way. If any equipment appears damaged, file a damage claim with the-0 carrier who delivered it.

Decide on a suitable location for the rack unit that will hold the server. It should be situated in a clean, dust-free area that is well ventilated. Avoid areas where heat, electrical noise and electromagnetic fields are generated. It will also require a grounded AC power outlet nearby. Be sure to read the precautions and considerations noted in Appendix A.

2.3 Preparing for Setup

The box in which the system was shipped should include the rackmount hardware needed to install it into the rack. Please read this section in its entirety before you begin the installation.

Choosing a Setup Location

• The system should be situated in a clean, dust-free area that is well ventilated. Avoid areas where heat, electrical noise and electromagnetic fields are generated.
• Leave enough clearance in front of the rack so that you can open the front door completely (\~25 inches) and approximately 30 inches of clearance in the back of the rack to allow sufficient space for airflow and access when servicing.
• This product should be installed only in a Restricted Access Location (dedicated equipment rooms, service closets, etc.).

- This product is not suitable for use with visual display workplace devices according to §2 of the German Ordinance for Work with Visual Display Units.

Rack Precautions

• Ensure that the leveling jacks on the bottom of the rack are extended to the floor so that the full weight of the rack rests on them.
• In single rack installations, stabilizers should be attached to the rack. In multiple rack installations, the racks should be coupled together.
• Always make sure the rack is stable before extending a server or other component from the rack.
• You should extend only one server or component at a time - extending two or more simultaneously may cause the rack to become unstable.

Server Precautions

• Review the electrical and general safety precautions in Appendix A.
• Determine the placement of each component in the rack before you install the rails.
• Install the heaviest server components at the bottom of the rack first and then work your way up.
• Use a regulating uninterruptible power supply (UPS) to protect the server from power surges and voltage spikes and to keep your system operating in case of a power failure.
• Allow any drives and power supply modules to cool before touching them.
• When not servicing, always keep the front door of the rack and all covers/panels on the servers closed to maintain proper cooling.

Rack Mounting Considerations

Ambient Operating Temperature

If installed in a closed or multi-unit rack assembly, the ambient operating temperature of the rack environment may be greater than the room's ambient temperature. Therefore, consideration should be given to installing the equipment in an environment compatible with the manufacturer's maximum rated ambient temperature (TMRA).

Airflow

Equipment should be mounted into a rack so that the amount of airflow required for safe operation is not compromised.

Mechanical Loading

Equipment should be mounted into a rack so that a hazardous condition does not arise due to uneven mechanical loading.

Circuit Overloading

Consideration should be given to the connection of the equipment to the power supply circuitry and the effect that any possible overloading of circuits might have on overcurrent protection and power supply wiring. Appropriate consideration of equipment nameplate ratings should be used when addressing this concern.

Reliable Ground

A reliable ground must be maintained at all times. To ensure this, the rack itself should be grounded. Particular attention should be given to power supply connections other than the direct connections to the branch circuit (i.e. the use of power strips, etc.).

To prevent bodily injury when mounting or servicing this unit in a rack, you must take special precautions to ensure that the system remains stable. The following guidelines are provided to ensure your safety:

• This unit should be mounted at the bottom of the rack if it is the only unit in the rack.
• When mounting this unit in a partially filled rack, load the rack from the bottom to the top with the heaviest component at the bottom of the rack.
• If the rack is provided with stabilizing devices, install the stabilizers before mounting or servicing the unit in the rack.
• Slide rail mounted equipment is not to be used as a shelf or a work space.

Slide rail mounted equipment is not to be used as a shelf or a work space.

Warning: do not pick up the server with the front handles. They are designed to pull the system from a rack only.

2.4 Installing the Rails

There are a variety of rack units on the market, which may require a slightly different assembly procedure. Do not use a two post "telco" type rack. This rail set fits a rack between 26.5" and 36.4" deep.

The following is a basic guideline for installing the system into a rack with the rack mounting hardware provided. You should also refer to the installation instructions that came with the specific rack you are using.

Installing the Rails onto the Rack

The front and rear ends of each rail have a locking latch. This latch is used to attach the rails to the rack.

To mount the rails onto the rack, first extend them by releasing the inner rails from the outer rails.

1. Using a screwdriver, loosen the screws holding the inner rail in place inside the outer rail.

Locking Latch

Figure 2-1. Releasing the Inner Rail

1. Mount the front end of the rail by pushing the outer rail latch through one of the square holes on the front of the rack.
2. Pull the inner rail out of the outer rail until it reaches the rear of the rack.
3. Mount the rear end of the rail by pushing the inner rail latch through one of the square holes on the rear of the rack. Take care to use the proper holes so that the rail is level.

Technical diagram of a server rack with labeled components and exploded view, showing internal structure and assembly steps.

Figure 2-2. Extending and Mounting a Rail

Stability hazard. The rack stabilizing mechanism must be in place, or the rack must be bolted to the floor before you slide the unit out for servicing. Failure to stabilize the rack can cause the rack to tip over.

1. Secure the length of the extended rail by tightening the screws into the holes closer to the inner rail.

Technical line drawing of a mechanical device with labeled parts and an arrow indicating motion (no text or symbols present)

Figure 2-3. Extending the Rail

1. Repeat the preceding steps for the other rail, making sure it is mounted at the same height as the first installed rail so that they are parallel.

Technical line drawing of a modular shelving unit with metal racks and storage compartments (no text or symbols)

Figure 2-4. Rails Installed in a Rack

1. Secure the rails to the rack by installing screws into the square holes on the front and rear of the rack aligned with the rail holes.

Technical line drawing of a mechanical assembly with a tool and magnified detail view (no text or symbols)

Figure 2-5. Securing Rails to a Rack

2.5 Installing the Chassis into the Rack

After the rails are installed on the rack, the chassis can be installed in the rack. It is heavy and requires two to three people or a lift.

Installing the Chassis into a Rack

1. Align the sides of the chassis with the front of the rails.
2. Slide the chassis into the rails, keeping the pressure even on both sides.

Technical line drawing of a server rack with mounting bracket and vertical railings (no text or symbols)

Figure 2-6. Installing the Chassis into the Rack

Note: Figures are for illustrative purposes only. Your actual chassis may differ. Always install servers into racks from the bottom up.

1. Push the chassis all the way into the rear of the rack until side brackets on the front of the chassis touch the front of the rack.

2. Secure the front of the chassis to the front of the rack by tightening screws through the chassis bracket holes and the rack holes aligned with them.

Technical line drawing of a mechanical assembly with vertical supports and a central panel (no text or symbols)

Figure 2-7. Securing the Chassis to the Rack

Loading HDDs into the Chassis on a Rack

1. Press the release buttons on both of the front handles downward simultaneously and pull the chassis handles forward from the rack.

1 Release Buttons

Figure 2-8. Opening the Chassis on a Rack

1. Pull open the chassis so that a locking lever on each side of the chassis aligns with the front of the chassis cover and front of the rack, then press the lever down on each side to lock the open part of the chassis in place.

Locking Lever

Figure 2-9. Locking in Place the Open Chassis on a Rack

1. HDDs and their carriers can be loaded into the chassis, or removed from the chassis by pressing their release buttons to eject their handles and then pulling the drives out by the handles.

Release Button

Figure 2-10. Loading HDDs in the Chassis in a Rack

Chapter 3

Maintenance and Component Installation

This chapter provides instructions on installing and replacing main system components. To prevent compatibility issues, only use components that match the specifications and/or part numbers given.

Installation or replacement of most components require that power first be removed from the system. Please follow the procedures given in each section.

3.1 Removing Power

Use the following procedure to ensure that power has been removed from the system. This step is necessary when removing or installing non hot-swap components or when replacing a non-redundant power supply.

1. Use the operating system to power down the system.
2. After the system has completely shut-down, disconnect the AC power cord(s) from the power strip or outlet. (If your system has more than one power supply, remove the AC power cords from all power supply modules.)
3. Disconnect the power cord(s) from the power supply module(s).

3.2 Accessing the System

Drive bays can be accessed by pulling the HDD drawer forward out of the chassis. Other components can be accessed though the front or rear of the chassis.

Caution: Except for short periods of time, do not operate the server without the cover in place. It helps provide proper airflow and prevent overheating.

HDD Drawer Access

The drives can be accessed by opening the drawer that contains them.

Opening the HDD Drawer

1. A locking handle is on either side of the chassis front. To unlock the drawer, press the release buttons on the front handles down into the unlocked position.

Technical line drawing of a mechanical assembly with an inset showing a close-up of a component (no text or symbols present)

Figure 3-1. Unlocking the Drawer

1. Simultaneously pull both handles forward, sliding the HDD drawer out of the storage enclosure.

Technical line drawing of a mechanical assembly with conveyor belt and housing (no text or symbols)

Figure 3-2. Opening the Drawer

3.3 Static-Sensitive Devices

Electrostatic Discharge (ESD) can damage electronic components. To avoid damaging your motherboard, it is important to handle it very carefully. The following measures are generally sufficient to protect the system PCBs from ESD.

Precautions

• Use a grounded wrist strap designed to prevent static discharge.
• Touch a grounded metal object before removing any PCB (printed circuit board) from its antistatic bag.
• Handle PCBs by their edges only; do not touch its components, peripheral chips, memory modules or gold contacts.
• When handling chips or modules, avoid touching their pins.
• Put the PCBs back into their antistatic bags when not in use.
• Use only the correct type of onboard CMOS battery. Do not install the onboard battery upside down to avoid possible explosion.

3.4 Processor and Heatsink Installation

The processor (CPU) must first be attached to the processor carrier to form the processor carrier assembly. This assembly gets attached to the heatsink to form the processor heatsink module (PHM), which is then installed into the CPU socket. Before installing, be sure to perform the steps below:

• Please carefully follow the instructions given on ESD precautions.
  • After shutting down the system, unplug the AC power cords from all power supplies.
• Check that the plastic protective cover is on the CPU socket and that none of the socket pins are bent. If they are, contact your retailer.
• When handling the processor, avoid touching or placing direct pressure on the LGA lands (gold contacts). Improper installation or socket misalignment can cause serious damage to the processor or the socket, and may require manufacturer repairs.
  • Thermal grease is pre-applied on new heatsinks. No additional thermal grease is needed.
  • Refer to the Supermicro website for updates on processor and memory support.
• All graphics in this manual are for illustration only. Your components may look different.

The Processor Carrier Assembly

The processor carrier assembly is comprised of the processor and the processor carrier.

To create the processor carrier assembly, please follow the steps below:

Note: Before installation, be sure to review the Static-Sensitive Devices section earlier in this chapter.

1. Hold the processor with the gold pins (LGA lands) facing down. Locate the gold triangle at the corner of the processor and the corresponding hollowed triangle on the processor carrier as shown below. These triangles indicate the location of pin 1.

2. Turn the processor over (with the gold pins up). Locate the CPU keys on the processor and the four latches on the carrier as shown below.

(with Processor Seated inside the Carrier)

1. Locate the lever on the carrier and press it down as shown below.

Technical diagram showing a mechanical assembly with an inset highlighting a lever mechanism, labeled 'Lever'

1. Using pin 1 as a guide, carefully align the CPU keys on the processor (A & B) with those on the carrier (a & b) as shown below.

CPU Key (on the processor) CPU Key (on the carrier) Latch B 1 A 2 a Latch CPU Key (on the processor) CPU Key (on the carrier)

1. Once aligned, carefully place one end of the processor under latch 1 on the carrier, and then press the other end down until it snaps into latch 2.

2. After the processor is placed inside the carrier, examine the four sides of the processor, making sure that the processor is properly seated on the carrier.

Processor Carrier Assembly (Top Side View) Processor Carrier Assembly

The Processor Heatsink Module (PHM)

After creating the processor carrier assembly, follow the instructions below to mount the heatsink onto the carrier assembly to form the processor heatsink module (PHM).

Note: If this is a new heatsink, the thermal grease has been pre-applied. Otherwise, apply the proper amount of thermal grease to the underside of the heatsink.

1. Turn the heatsink over with the thermal grease facing up. Pay attention to the two triangle cutouts (A, B) located at the diagonal corners of the heatsink as shown in the drawing below.

2. Hold the processor carrier assembly upside-down to locate the triangles on the processor and the carrier, which indicate pin 1.

3. Turn the processor carrier assembly over so that the gold pins are facing up. Locate the two pin 1 locations ("A on the processor and "a" on the processor carrier assembly).

4. Align "a" on the processor carrier assembly with the triangular cutout "A" on the heatsink along with "b", "c", "d" on the processor assembly with "B", "C", "D" on the heatsink.

5. Once properly aligned, place the heatsink on the processor carrier assembly with all corners matched up, making sure that the four clips are properly securing the heatsink.

Processor Carrier Assembly Pin1 a b c D A B C Heatsink

Preparing the CPU Socket for Installation

The CPU socket comes with a plastic protective cover, which needs to be removed before installing the Processor Heatsink Module (PHM). Do this by gently squeezing the grip tabs then pulling the cover off.

CPU Socket with Plastic Protective Cover Grip Tab

Installing the PHM into the CPU Socket

After assembling the Processor Heatsink Module (PHM), you are ready to install it into the CPU socket.

1. Locate four threaded fasteners (a, b, c, d) on the CPU socket.

CPU Socket Threaded Fastener (a, b, c, d: Threaded Fasteners) CPU Socket Pin1

1. Locate four peek nuts (A, B. C. D) and four rotating wires (1, 2, 3, 4) on the heatsink as shown below.

Heatsink A, B, C, D: Peek Nut 1, 2, 3, 4: Rotating Wire a, b, c, d: Threaded Fastener Rotating Wire 4 Rotating Wire 3 Rotating Wire 1 Peek Nut CPU Socket d a Threaded Fastener Rotating Wire Peek Nut (Unlatched) (latched)

1. Check the rotating wires (1, 2, 3, 4) to make sure that they are in the unlatched position as shown.

Unlatched State Rotating Wire Side View Peak Nut Top View

1. Align peek nut "A" (next to the triangular pin 1 on the heatsink) with threaded fastener "a" on the CPU socket. Then align peek nuts "B", "C", "D" on the heatsink with threaded fasteners "b", "c", "d" on the CPU socket, making sure that all peek nuts and threaded fasteners are properly aligned.

2. Once aligned, gently place the heatsink on the CPU socket, making sure that each peek nut is properly attached to its corresponding threaded fastener.

A, B, C, D: Peek Nut on the Heatsink B D A C b c d a, b, c, d: Threaded Fastener on the CPU socket

1. Press all four rotating wires outward and make sure that the heatsink is securely latched into the CPU socket.

Rotating Wire Rotating Wire Latched State Top View

1. With a t30-bit screwdriver, tighten all peek nuts in the sequence of "A", "B", "C", and "D" with even pressure. To avoid damaging the processor or socket, do not use a force greater than 12 lbf-in when tightening the screws.

2. Examine all corners of the heatsink to ensure that the PHM is firmly attached to the CPU socket.

Technical diagram showing mechanical assembly with labeled components A, B, C, D and directional arrows indicating motion or force

Removing the PHM from the CPU Socket

Before removing the PHM from the motherboard, first shut down the system and unplug the AC power cord from all power supplies.

1. Use a t30-bit screwdriver to loosen the four peek nuts on the heatsink in the sequence of A, B, C, and D.

B C Peek Nut D A

1. Once the peek nuts have been loosened from the CPU socket, press the rotating wires inward to unlatch the PHM from the socket as shown below.

Technical illustration of a dual-chamber industrial cooling unit with cooling fins and heat exchangers (no text or symbols)

Unlatched State Rotating Wire Peak Nut Side View

1. Gently lift the PHM upward to remove it from the CPU socket.

Technical diagram of an electronic component with mounting base and internal structure, showing red directional arrows indicating assembly or movement (no text or symbols present)

Removing the Processor Carrier Assembly from the PHM

To remove the processor carrier assembly from the PHM, please follow the steps below:

1. Detach the four plastic clips (marked a, b, c, d) on the processor carrier assembly from the four corners of the heatsink (marked A, B, C, D) as shown below.

Processor Carrier Assembly Pin1 Pin1 a b c D A Heatsink B C

1. When all plastic clips have been detached from the heatsink, remove the processor carrier assembly from the heatsink

Technical diagram of an electronic component with red arrows indicating assembly or force direction (no text or symbols present)

Removing the Processor from the Processor Carrier Assembly

Once you have removed the processor carrier assembly from the PHM, you are ready to remove the processor from the processor carrier by following the steps below.

1. Unlock the lever from its locked position and push it upwards to disengage the processor from the processor carrier as shown below right.

Processor Carrier Assembly Lever

1. Once the processor has been loosened from the carrier, carefully remove the processor from the carrier.

Note: Please handle the processor with care to avoid damaging it or its pins.

Isometric technical diagram of a computer motherboard with a highlighted component and red arrow indicating upward motion (no text or symbols)

3.5 Memory

Memory Support

The X12DSC-6 motherboard in each node supports up to 4TB of 3DS LRDIMM/LRDIMM/3DS RDIMM/RDIMM DDR4 (288-pin) ECC memory with speeds of 3200/2933/2666 MHz in 16 memory slots and up to 4TB of Intel Optane PMem 200 Series with speeds of up to 3200 MHz. (See the notes below.)

Notes: Intel® Optane™ Persistent Memory (PMem) 200 Series are supported by the 3rd Gen Intel Xeon Scalable Platinum, Gold, and selected Silver processor.

P1-DIMMB2/P2-DIMMB2 memory slots are reserved for Intel Optane PMem 200 Series only.

General Memory Population Requirements

1. Be sure to use the memory modules of the same type and speed on the motherboard. Mixing of memory modules of different types and speeds is not allowed.
2. Using unbalanced memory topology such as populating two DIMMs in one channel while populating one DIMM in another channel will result in reduced memory performance.
3. Populating memory slots with a pair of DIMM modules of the same type and size will result in interleaved memory, which will improve memory performance.

Memory Support for 3rd Gen Intel Xeon Scalable Processors
Type	Ranks Per DIMM & Data Width	DIMM Capacity (GB)		Speed (MT/s); Voltage (V); Slots Per Channel (SPC) and DIMMs Per Channel (DPC)
				1DPC(1-DIMM Per Chan- nel)	2DPC(2-DIMM Per Channel)
		8Gb 16Gb	1.2 V 1.2 V
RDIMM	SRx8 8GB 16GB			3200 3200
	SRx4 16GB 32GB
	DRx8 16GB 32GB
	DRx4 32GB 64GB
RDIMM 3Ds (4R/8R) X4		2H- 64 GB4H-128 GB	2H- 128 GB4H-256 GB
LRDIMM QRx4 64GB 128GB		3200 3200
LRDIMM - 3Ds	(4R/8R) X4	4H-128 GB	2H- 128 GB4H-256 GB	3200 3200

Memory Population Table (with 16 Slots)
When 1 CPU is used: Memory Population Sequence
1 CPU & 1 DIMM CPU1: P1-DIMMA1
1 CPU & 2 DIMMs* CPU1: P1-DIMMA1/P1-DIMME1
1 CPU & 4 DIMMs* CPU1: P1-DIMMA1/P1-DIMME1/P1-DIMMC1/P1-DIMMG1
1 CPU & 6 DIMM CPU1: P1-DIMMA1/P1-DIMME1/P1-DIMMC1/P1-DIMMG1/P1-DIMMB1/P1-DIMMF1
1 CPU & 8 DIMMs*	CPU1: P1-DIMMA1/P1-DIMME1/P1-DIMMC1/P1-DIMMG1/P1-DIMMB1/P1-DIMMF1/P1-DIMMD1/P1-DIMMH1
When 2 CPUs are used: Memory Population Sequence
2 CPUs & 2 DIMMs*	CPU1: P1-DIMMA1CPU2: P2-DIMMA1
2 CPUs & 4 DIMMs*	CPU1: P1-DIMMA1/P1-DIMME1CPU2: P2-DIMMA1/P2-DIMME1
2 CPUs & 6 DIMMs	CPU1: P1-DIMMA1/P1-DIMME1/P1-DIMMC1/P1-DIMMG1CPU2: P2-DIMMA1/P2-DIMME1
2 CPUs & 8 DIMMs*	CPU1: P1-DIMMA1/P1-DIMME1/P1-DIMMC1/P1-DIMMG1CPU2: P2-DIMMA1/P2-DIMME1/P2-DIMMC1/P2-DIMMG1
2 CPUs & 10 DIMMs	CPU1: P1-DIMMA1/P1-DIMME1/P1-DIMMC1/P1-DIMMG1/P1-DIMMB1/P1-DIMMF1CPU2: P2-DIMMA1/P2-DIMME1/P2-DIMMC1/P2-DIMMG1
2 CPUs & 12 DIMMs*	CPU1: P1-DIMMA1/P1-DIMME1/P1-DIMMC1/P1-DIMMG1/P1-DIMMB1/P1-DIMMF1CPU2: P2-DIMMA1/P2-DIMME1/P2-DIMMC1/P2-DIMMG1/P2-DIMMB1/P2-DIMMF1
2 CPUs & 14 DIMMs	CPU1: P1-DIMMA1/P1-DIMME1/P1-DIMMC1/P1-DIMMG1/P1-DIMMB1/P1-DIMMF1/P1-DIMMD1/P1-DIMMH1CPU2: P2-DIMMA1/P2-DIMME1/P2-DIMMC1/P2-DIMMG1/P2-DIMMB1/P2-DIMMF1
2 CPUs & 16 DIMMs*	CPU1: P1-DIMMA1/P1-DIMME1/P1-DIMMC1/P1-DIMMG1/P1-DIMMB1/P1-DIMMF1/P1-DIMMD1/P1-DIMMH1CPU2: P2-DIMMA1/P2-DIMME1/P2-DIMMC1/P2-DIMMG1/P2-DIMMB1/P2-DIMMF1/P2-DIMMH1

Note: To maximize memory performance, please use the memory configurations marked with "*" above as these configurations are recommended by Supermicro for optimal memory performance

PMem 200 Series Population table for X12DP Motherboards (w/16 Slots)

Note: Intel® Optane™ Persistent Memory (PMem) 200 Series are supported by 3rd Gen Intel Xeon Scalable Platinum, Gold, and selected Silver processors.

16-DIMM Motherboard PMem Population within 1 CPU socket
DDR4+PMem	Mode AD	Interleave	P1-DIMMF1	P1-DIMME1	P1-DIMMH1	P1-DIMMG1	P1-DIMMC1	P1-DIMMD1	P1-DIMMA1	P1-DIMMB1
4+4	ADMM	One - x4	PMem	DDR4	PMem	DDR4	DDR4	PMem	DDR4	PMem
		One - x4	DDR4	PMem	DDR4	PMem	PMem	DDR4	PMem	DDR4
6+1	AD	One - x1	DDR4	DDR4	-	DDR4	DDR4	PMem	DDR4	DDR4
			-	DDR4	DDR4	DDR4	DDR4	DDR4	DDR4	PMem
			DDR4	DDR4	PMem	DDR4	DDR4	-	DDR4	DDR4
			PMem	DDR4	DDR4	DDR4	DDR4	DDR4	DDR4	-
			DDR4	DDR4	DDR4	-	PMem	DDR4	DDR4	DDR4
			DDR4	-	DDR4	DDR4	DDR4	DDR4	PMem	DDR4
			DDR4	DDR4	DDR4	PMem	-	DDR4	DDR4	DDR4
			DDR4	PMem	DDR4	DDR4	DDR4	DDR4	-	DDR4

Legend

DDR4 Type and Capacity

DDR4 See Validation Matrix (DDR4 DIMMs validated with PMem)

Capacity

PMem Any Capacity (Uniformly for all channels for a given configuration)

• Mode definitions: AD = App Direct Mode, MM = Memory Mode.
• No mixing of PMem and NVDIMMs within the platform.
• For MM, NM/FM ratio is between 1:4 and 1:16. (NM = Near Memory (DRAM); FM = Far Memory (PMem)).
• Matrix targets configs for optimized PMem to DRAM cache ratio in MM mode.
• For each individual population, different PMem rearrangements among channels are permitted so long as the configuration doesn't break X12 DP memory population rules.
• Ensure the same DDR4 DIMM type and capacity are used for each DDR4 + PMem population.
• If the system detects an unvalidated configuration, then the system issues a BIOS warning. The CLI functionality is limited in non-POR configurations, and select commands will not be supported.

Validation Matrix (DDR4 DIMMs with PMem 200 Series)
DIMM Type	Ranks Per DIMM & Data Width (Stack)	DIMM Capacity (GB)
		DRAM Density
		8Gb 16Gb
RDIMM (up to 3200)	1Rx8 N/A N/A
	1Rx4 16GB 32GB
	1Rx8 16GB 32GB
	1Rx4 32GB 64GB
RDIMM 3DS (up to 3200)	4Rx4 (2H) N/A 128GB
	8Rx4 (4H) NA 256GB
LRDIMM (up to 3200) 4Rx4 64GB 128GB
LRDIMM 3DS (up to 3200)	4Rx4 (2H) N/A N/A
	8Rx4 (4H)	128GB	256GB

DIMM Installation

1. Insert the desired number of DIMMs into the slots based on the recommended DIMM population tables shown above.
2. Push the release tabs on both ends of the DIMM slot outwards to unlock it.

1. Align the key of the DIMM module with the receptive point on the memory slot.

Key

1. Align the notches on both ends of the module with the receptive points on the ends of the slot.

Notches

1. Push both ends of the module straight down into the slot until the module snaps into place.

Push both ends straight down into the memory slot.

Illustration of two hands holding a mechanical component with blue directional arrows indicating motion (no text or symbols)

1. Press the release tabs to the lock positions to secure the DIMM module into the slot.

DIMM Removal

Press both release tabs on the ends of the DIMM module to unlock it. Once the DIMM module is loose, remove it from the memory slot.

Warning! To avoid causing any damage to the DIMM module or the DIMM socket, do not use excessive force when pressing the release tabs on the ends of the DIMM socket. Handle DIMMs with care. Be aware and follow the ESD instructions given at he beginning of this chapter.

3.6 Mezzanine Card Installation

For SAS3 support, be sure to follow the instructions below to install the mezzanine card in the JMCN1, JMCN2, and JMCN3 connectors on the motherboard.

AOM-SADPT-S

The Supermicro AOM-SADPT-S is an optional mezzanine card that provides a by-pass connection to SAS devices for the SAS controller in the PCI slot. This hosts connectivity to two NVME M.2 devices (2280/22110 form factor) and PCIe NTB switch PM40036.

P-b CODE 2-104A2-M0A 35 L1 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 R13 R14 R15 R16 R17 R18 R19 R20 R21 R22 R23 R24 R25 R26 R27 R28 R29 R30 R31 R32 R33 R34 R35 R36 R37 R38 R39 R40 R41 R42 R43 R44 R45 R46 R47 R48 R49 R50

AOM-S3616

The Supermicro AOM-S3616-S is a mezzanine card that hosts two SAS 3616 controllers ideal for high-performing, bandwidth-intensive applications such as video streaming, medical imaging and big data analytics. For driver, firmware and management tools please refer to the system page on the Supermicro website. This card hosts connectivity to two NVME M.2 devices (2280/22110mm form factor), PCIe NTB switch PM40036, and Ethernet controller for HA heartbeat functionality.

NMM/M_2_2 SAS CODE SAS CODE MAC CODE SUPRE® AOM-S3616-S DESIGNED IN USA PC:21.01

Installing the Mezzanine Card

1. Power down the server and extend the node to which you want the card installed from the chassis.
2. Align the mezzanine card with the mezzanine tray.
3. Secure the mezzanine board to the tray with screws as shown below.
4. Flip the assembly over. Make sure the left and right latches are in the unlocked position, then align the tray to its position on the motherboard and lower it into place.
5. After making sure it is fully seated into the motherboard, rotate the latches forward to their locked position.
6. Carefully push down on the middle section to ensure the connectors are fully seated.
7. Push the node back into the chassis.

Latch Latch

Figure 3-3. Installing the Mezzanine Card

3.7 Motherboard Battery

The motherboard uses non-volatile memory to retain system information when system power is removed. This memory is powered by a lithium battery residing on the motherboard.

Replacing the Battery

Begin by removing power from the system.

1. Push aside the small clamp that covers the edge of the battery. When the battery is released, lift it out of the holder.
2. To insert a new battery, slide one edge under the lip of the holder with the positive (+) side facing up. Then push the other side down until the clamp snaps over it.

Note: Handle used batteries carefully. Do not damage the battery in any way; a damaged battery may release hazardous materials into the environment. Do not discard a used battery in the garbage or a public landfill. Please comply with the regulations set up by your local hazardous waste management agency to dispose of your used battery properly.

LITHIUM BATTERY BATTERY HOLDER

Figure 3-4. Installing the Onboard Battery

Warning: There is a danger of explosion if the onboard battery is installed upside down (which reverses its polarities). This battery must be replaced only with the same or an equivalent type recommended by the manufacturer (CR2032).

3.8 Storage Drives

The SSG-640SP-DE1CR60 supports 60 x3.5" hard drives. The drives can be removed without powering down the system if your operating system supports RAID. In addition, each node supports two M.2 NVMe drives, two internal 2.5" SATA drives, and optional two 7mm or one 15mm internal NVMe drives.

Hard Drive Carrier Indicators

Each hard drive carrier has two LED indicators: an activity indicator and a status indicator. The status indicator functions in RAID configurations. For non-RAID configurations, it remains off. See the table below for details.

Hard Drive Carrier LED Indicators
	Color Blinking Pattern Behavior for Device
Activity LED	Blue Solid on Indicates a SAS drive
	Blue Off Indicates a SATA drive
	Blue Blinking Drive is actively being accessed
Status LED	Red Solid on Drive failed
	Red Blinking at 1Hz RAID is rebuilding
	Red Blinking at 3Hz Indicates a hot spare
	Red Blinking at 4Hz Locates a drive
	Red Off Idle

Note: Enterprise level hard disk drives are recommended for use in Supermicro chassis and servers. For information on recommended HDDs, visit the Supermicro website and check the "Drive Options" in the product webpage.

Drive Carrier LEDs

Figure 3-5. Hard Drive Carrier Indicators

Installing Hot-Swap 3.5" Hard Drives

The hard drives use tool-less drive carriers to simplify their installation into the drive trays.

Removing Hard Drive Carriers from the Chassis

1. Power down the system as described in Section 3.1 and open the chassis cover as described in Section 3.2.
2. Remove hard drive carriers beginning from the center of the storage chassis and then working outward.
3. Slide the release button on the drive carrier to open the handle.
4. Use the handle to pull the drive carrier up and out of the chassis.

Technical line drawing of a server rack with internal components and an inset view showing a mechanical assembly (no text or symbols)

Figure 3-6. Removing a 3.5" Drive Carrier

Note: Enterprise level hard disk drives are recommended for use in Supermicro chassis and servers. For information on recommended HDDs, visit the Supermicro website at http://www.supermicro.com/products/nfo/files/storage/SBB-HDDCompList.pdf

Hard Drive Installation

The drives are mounted in drive carriers to simplify their installation and removal from the chassis. These carriers also help promote proper airflow through the drive bays.

Removing Hard Drive Carriers from the Chassis

1. Pull the storage enclosure forward out of the chassis.
2. Slide the release button on the drive carrier, which opens the carrier handle.
3. Use the drive carrier handle to pull the drive out of the chassis.

Carrier Breakout Lever release Pull Out Side of Carrier Alignment Pins Notch for Connectors

Figure 3-7. Opening the Drive Carrier

Caution: Except for short periods of time, such as swapping hard drives, do not operate the server with the hard drive bays empty.

Installing a 3.5" Hard Disk Drive

With the drive carrier removed from the storage compartment:

1. Under the main the carrier handle, find and the lift the breakout lever and pull out the side of the carrier.

2. Remove the dummy drive from the carrier.

3. Insert the hard drive into the drive carrier. Orient the drive by matching the two alignment pins on the side and by noting the notch in the carrier for the HDD connectors. Close the side of the carrier until it snaps in place.

4. Slide the carrier assembly into its spot in the chassis until it clicks into locked position.

Expander Module Replacement

Each expander module displays a green-lit LED when operating normally. If the LED is red or flashing, it indicates a failure and a replacement expander module should be installed.

Each expander module has two latches on its top side, which lock the expander into place and must be opened to remove the expander from the chassis. Pull both latches up to the open position, as shown below, then lift the unlocked expander module out of the chassis.

Technical line drawing of a server rack with grid panels and ventilation slots, showing internal structure and component assembly (no text or symbols)

Figure 3-8. Replacing an Expander Module

3.9 System Cooling

Fans

Six hot-swap, heavy-duty rear mounted fans provide cooling. They can be replaced without powering down the system.

Fan speed is controlled by a system temperature setting in BMC. If a fan fails, the remaining fans will ramp up to full speed. Replace any failed fan at your earliest convenience with the same type and model. The system can continue to run with a failed fan.

Replacing a System Fan

1. While the power is running, examine the fans to determine which fan has failed.
2. On the failed fan, simultaneously squeeze the fan's side release tabs inward.
3. Pull the fan out of the motherboard sled using the tabs.

Caution: Fans will continue to rotate for a brief time after removing them from the chassis. To avoid injury, keep fingers clear of the rotating fan blades.

TOP Release Tabs

Figure 3-9. Fan Release Tabs

Release Tab

Figure 3-10. Removing a Fan or Drive from a Motherboard Sled

1. Place the new fan into the vacant fan bay and confirm that the fan is fully seated in the fan bay.
2. Make sure the new fan is functioning properly.

Checking the Server Air Flow

• Make sure there are no objects to obstruct airflow in and out of the server.
• If you are using a front bezel, make sure the bezel filter is replaced periodically.
• Do not operate the server without drives or drive trays in the drive bays.
• Use only recommended server parts.
• Make sure no wires or foreign objects obstruct air flow through the chassis. Pull all excess cabling out of the airflow path or use shorter cables.

The control panel LEDs display system heat status. See “Control Panel” in Chapter 1 for details.

Overheating

There are several possible responses if the system overheats.

Responses

If the enclosure overheats:

1. Use the LEDs to determine the nature of the overheating condition.
2. Confirm that the chassis covers are installed properly.
3. Make sure all fans are present and operating normally.
4. Check the routing of the cables.

3.10 Power Supply

The system features redundant power supplies and will continue to operate if one module fails, when it should be replaced as soon as convenient. The modules are hot-swappable, meaning they can be changed without powering down the system. New units can be ordered directly from Supermicro or authorized distributors. The power supplies are auto-switching capable. This feature enables them to automatically sense the input voltage and operate at a 100-120V or 180-240V range (200-240V for the 2600W power supplies).

Power Supply LEDs

On the rear of the power supply module, an LED displays the status.

• Solid Green: When illuminated, indicates that the power supply is on.
• Solid Amber: When illuminated, indicates the power supply is plugged in and turned off, or the system is off but in an abnormal state.
• Blinking Amber: When blinking, this system power supply temperature has reached 63C. The system will automatically power-down when the power supply temperature reaches 70C and restart when the power supply temperature goes below 60C.

Changing the Power Supply Module:

1. Unplug the AC cord from the module to be replaced.
2. On the back of the module, push the release tab sideways, as illustrated.
3. Pull the module out using the handle.
4. Push the new module into the power bay until it clicks. Replace with the same model.
5. Plug the AC power cord back into the module.

Release Tab

Figure 3-11. Removing the Power Supply

3.11 PCI Expansion Cards

The system accepts up to three PCIe expansion cards on each node.

Front view of a server rack with hexagonal heat sinks and labeled CPU ports (no readable text beyond labels)

Figure 3-12. Expansion Slots

Note: The bottom tier is Node A and the top tier is Node B.

Expansion Card Chassis Slots (per Node)

Item Description

1 PCIe 4.0x16 low-profile slot

2 PCIe 4.0x16 low-profile slot

3 PCIe 4.0x16 low-profile slot

Notes: HHHL = half height, half length.

Full height = 4.2", low profile = 2.5", full length = 10.5", half length = 6.6"

Installing an Expansion Card

1. Power down the system as described in section 3.1 and remove the cover.
2. Unscrew and remove the chassis slot cover.
3. Insert the expansion card into a slot on the motherboard while aligning the expansion card backplate with the open slot in the rear of the chassis. Secure with a screw.
4. Replace the cover and power.

3.12 BMC

The BMC can be reset using the button on the front control panel or on the chassis rear.

• Reset—Press and hold the button. After six seconds, the LED blinks at 2 Hz. The BMC resets and the reset duration is \~250 ms. Then the BMC starts to boot.
• Restore factory default configuration—Hold the button for twelve seconds. The LED blinks at 4 Hz while defaults are configured.
• Firmware update—the UID LED blinks at 10Hz during a firmware update.

BMC Reset Options
Event LED (Green)
Reset Blinks at 2 Hz
Restore Defaults	Blinks at 4 Hz
Update Blinks at 10 Hz

Chapter 4

Motherboard Connections

This section describes the connections on the motherboard and provides pinout definitions. Note that depending on how the system is configured, not all connections are required. The LEDs on the motherboard are also described here. A motherboard layout indicating component locations may be found in Chapter 1. More detail can be found in the Motherboard Manual Please review the Safety Precautions in Appendix A before installing or removing components.

4.1 Power Connections

Backplane Connectors

BCN1 and BCN2 connect to the BPN-SAS3-947SB backplane connectors to provide power to the motherboard.

4.2 Headers and Connectors

Fan Headers

Three 4-pin (JFAN, FAN1 and FAN2) and one 8-pin (JFANPW) fan headers are located on the motherboard to provide CPU and system cooling.

TPM/Port 80 Header

A Trusted Platform Module (TPM)/Port 80 header is located at JTPM1 to provide TPM support and Port 80 connection. Use this header to enhance system performance and data security. Refer to the table below for pin definitions. Please go to the following link for more information on the TPM: http://www.supermicro.com/manuals/other/TPM.pdf.

Trusted Platform Module/Port 80 Header Pin Definitions
Pin# Definition Pin# Definition
1 P3V3	2 SPI_TPM_CS_N
3 PCIE_RESET_N# 4 SPI_PCH_MISO
5 SPI_PCH_CLK# 6 Ground
7 SPI_PCH_MOSI 8 N/A
9 JTPM1_P3V3A 10 IRQ_TPM_SPIN_N

VROC RAID Key Header

A VROC RAID Key header is located at JRK1 on the motherboard. Install a VROC RAID Key on JRK1 for NVMe RAID support as shown in the illustration below. Please refer to the layout below for the location of JRK1.

Intel VROC KeyPin Definitions
Pin#	Definition
1	Ground
2	3.3V Standby
3	Ground
4	PCH RAID Key

VROC Key VROC Key Header (JRK1)

Note: The graphics contained in this user's manual are for illustration purposes only. The components installed in your system may or may not look exactly the same as the graphics shown in the manual.

Chassis Intrusion

A Chassis Intrusion header is located at JL1 on the motherboard. Connect an appropriate cable from JL1 to the chassis so that you can be informed of a chassis intrusion (via BMC) when the system case is opened. Refer to the table below for pin definitions.

Chassis Intrusion Pin Definitions

Pin# Definition

1 Intrusion Input

2 Ground

Mezzanine Board Docking Connectors

Mezzanine Board connectors are located at JMCN1, JMCN2 and JMCN3. This mezzanine board also supports M.2 NVMe devices in 2280 and 22110 form factors. M.2, formerly known as Next Generation Form Factor (NGFF), replaces mini PCIe devices, allowing for a variety of card sizes, increased functionality, and spatial efficiency.

4.3 Input/Output Ports

Rear I/O Ports

See the figure below for the locations and descriptions of the I/O ports on the rear of the motherboard.

Diagram of two Ethernet ports with numbered labels pointing to different connection points.

Rear I/O Ports
# Description # Description
1 LAN1 5 Dedicated BMC_LAN
2 LAN2 6 UID LED
3 USB0 7 UID Switch
4 USB1 8

VGA Connections

There are two VGA connections on the X12DSC-6. The rear VGA port is located at JVGA1 on the rear I/O panel, and the front VGA header is located at FP2 on the motherboard. These VGA connections provide analog interface support between the computer and the video displays.

COM Port

A COM port that supports a serial link interface is included on the rear I/O panel.

LAN Ports

Two LAN ports (LAN1, LAN2) are located on the rear I/O back panel. In addition, a dedicated BMC_LAN Port is located above the USB0/1 ports. All of these ports accept RJ45 cables. Please refer to the LED Indicator section for LAN LED information.

LAN PortPin Definitions
Pin# Definition Pin# Definition
1 TD0-	11 P3V3_Dual
2 TD0+	12 Act LED (Yellow)
3 TD1-	13		Link 1000(Amber)
4 TD1+	14		Link 100 LED(Green)
5 TD2-	15 GND
6 TD2+	16 GND
7 TD3-	17 GND
8 TD3+	18 GND
9 COMMCT
10 GND

BMC LANPin Definitions
Pin# Definition Pin# Definition
9 19 GND
10 TD0+ 20		Act LED(Yellow)
11 TD0- 21		Link 100 LED(Green)
12 TD1+ 22		Link 1000 LED(Amber)
13 TD1- 23 SGND
14 TD2+ 24 SGND
15 TD2- 25 SGND
16 TD3+ 26 SGND
17 TD3-
18 GND

Universal Serial Bus (USB) Ports

There are two USB 3.0 ports (USB0/1) located on the I/O back panel. Refer to the board layout below for the location.

Back Panel USB 0/1 (3.0) Pin Definitions
Pin#	Definition	Pin#	Definition
A1	VBUS	B1	Power
A2	D-	B2	USB_N
A3	D+	B3	USB_P
A4	GND	B4	GND
A5	Stda_SSRX-	B5	USB3_RN
A6	Stda_SSRX+ B6	USB3_RP
A7	GND	B7	GND
A8	Stda_SSTX-	B8	USB3_TN
A9	Stda_SSTX+	B9	USB3_TP

4.4 Jumpers

Explanation of Jumpers

To modify the operation of the motherboard, jumpers are used to choose between optional settings. Jumpers create shorts between two pins to change the function associated with it. Pin 1 is identified with a square solder pad on the printed circuit board. See the motherboard layout page for jumper locations.

Note: On a two-pin jumper, "Closed" means the jumper is on both pins and "Open" indicates the jumper is either on only one pin or has been completely removed.

Connector Pins Jumper Setting 3 2 1 3 2 1

CMOS Clear

JBT1 is used to clear CMOS, which will also clear any passwords. Instead of pins, this jumper consists of contact pads to prevent accidentally clearing the contents of CMOS.

To Clear CMOS

1. First power down the system and unplug the power cord(s).
2. Remove the cover of the chassis to access the motherboard and remove the battery from the motherboard.
3. Short the CMOS pads with a metal object such as a small screwdriver for at least four seconds.
4. Remove the screwdriver (or shorting device).
5. Replace the cover, reconnect the power cord(s), and power on the system.

Note: Clearing CMOS will also clear all passwords. Do not use the PW_ON connector to clear CMOS.

Manufacturing Mode Select

Close JPME2 to bypass SPI flash security and force the system to use Manufacturing Mode, which will allow you to flash the system firmware from a host server to modify system settings. See the table below for jumper settings.

Manufacturing ModeJumper Settings
Jumper Setting Definition
Pins 1-2 Normal
Pins 2-3 Manufacturing Mode

Watch Dog

JWD1 controls the Watch Dog function. Watch Dog is a monitor that can reboot the system when a software application hangs. Jumping pins 1-2 will cause Watch Dog to reset the system if an application hangs. Jumping pins 2-3 will generate a non-maskable interrupt signal for the application that hangs. Watch Dog must also be enabled in BIOS. The default setting is Reset.

Note: When Watch Dog is enabled, the user needs to write their own application software to disable it.

Watch DogJumper Settings
Jumper Setting Definition
Pins 1-2 Reset (Default)
Pins 2-3 NMI
Open Disabled

4.5 LED Indicators

LAN LEDs

Two LAN ports (LAN 1 and LAN 2) are located on the I/O back panel of the motherboard. Each Ethernet LAN port has two LEDs. The green LED indicates activity, while the other Link LED may be green, amber, or off to indicate the speed of the connection. Refer to the tables below for more information.

LAN1/2 Activity LED (Right)LED State
Color Status Definition
Green Flashing Active

LAN1/2 Link LED (Left)LED States
LED Color Definition
Green 10Gbps
Yellow/Amber 1Gbps

Unit ID LED

A rear UID LED indicator (UID-LED) is located near the UID switch on the I/O back panel. This UID indicator provides easy identification of a system unit that may need service.

UID LEDLED State
LED Color Definition
Blue: On Unit	Identified

BMC\_LAN LEDs

In addition to LAN1 and LAN2, a BMC LAN is also located on the I/O back panel. The amber LED on the right indicates activity, while the LED on the left indicates the speed of the connection. Refer to the table below for more information.

BMC LAN LEDs LED States
Color/State Definition
Link (left)	Green: Solid Amber: Solid	100 Mbps 1Gbps
Activity (Right)	Amber: Blinking Active

Onboard Power LED

The Onboard Power LED is located at LE3 on the motherboard. When this LED is on, the system is powered on. Be sure to turn off the system and unplug the power cord before removing or installing components. Refer to the table below for more information.

Onboard Power LED LED States
LED Color Definition
Off	System Off(power cable not connected)
Green System	On

BMC Heartbeat LED

A BMC Heartbeat LED is located at LEDM1 on the motherboard. When LEDM1 is blinking, the BMC is functioning normally. Refer to the table below for more information.

BMC Heartbeat LEDLED States
LED Color Definition
Green:Blinking	BMC Normal

4.6 Storage Ports

SATA 3.0 Ports

The X12DSC-6 has two SATA 3.0 connectors (S-SATA4/S-SATA5) on the motherboard. Two extra SATA ports are on the mezzanine board. These SATA ports are supported by the Intel® C621A chipset. S-SATA4 and S-SATA5 can be used with Supermicro SuperDOMs, which are yellow SATA DOM connectors with power pins built in, and do not require external power cables. Supermicro SuperDOMs are backward-compatible with regular SATA HDDs or SATA DOMs that need external power cables.

PCIe M.2 Slot & M.2-HC S-SATA4/5 Slots

A PCIe 4.0 M.2 slot (PCIe M.2) is located next to the TPM/Port80. In addition, two M.2-HC S-SATA slots (JMD1/JMD2) also support PCIe 4.0 x4 for NVMe/SATA Solid State Devices (SSDs). These M.2 slots support 2280 and 22110 form factors via SlimSAS x8 cables (required). The M.2 slots allow for a variety of card sizes with increased functionality and spatial efficiency.

Chapter 5

Software

After the hardware has been installed, you can install the Operating System (OS), configure RAID settings and install the drivers.

5.1 Microsoft Windows OS Installation

If you will be using RAID, you must configure RAID settings before installing the Windows OS and the RAID driver. Refer to the RAID Configuration User Guides posted on our website at www.supermicro.com/support/manuals.

Installing the OS

1. Create a method to access the MS Windows installation ISO file. That might be a DVD, perhaps using an external USB/SATA DVD drive, or a USB flash drive, or the IPMI KVM console.
2. Retrieve the proper RST/RSTe driver. Go to the Supermicro web page for your motherboard and click on "Download the Latest Drivers and Utilities", select the proper driver, and copy it to a USB flash drive.
3. Boot from a bootable device with Windows OS installation. You can see a bootable device list by pressing F11 during the system startup.

Please select boot device: ATEN Virtual CDROM YSOJ → IPMI virtual drive (Legacy) ASUS SDRW-08D2S-U F601 → USB DVD device (Legacy) USB FLASH DRIVE PMAP → USB flash drive with OS installation (Legacy) IBA 40-10G Slot 1900 v1060 → PXE boot (Legacy) UEFI: ATEN Virtual CDROM YSOJ → IPMI virtual drive (UEFI) UEFI: ASUS SDRW-08D2S-U F601 → USB DVD device (UEFI) UEFI: Built-in EFI Shell Enter Setup ↑ and ↓ to move selection ENTER to select boot device ESC to boot using defaults

Figure 5-1. Select Boot Device

1. During Windows Setup, continue to the dialog where you select the drives on which to install Windows. If the disk you want to use is not listed, click on "Load driver" link at the bottom left corner.

Where do you want to install Windows? Name Total size Free space Type Refresh Delete Format New Load driver Extend We couldn't find any drives. To get a storage driver, click Load driver. Next

Figure 5-2. Load Driver Link

To load the driver, browse the USB flash drive for the proper driver files.

• For RAID, choose the SATA/sSATA RAID driver indicated then choose the storage drive on which you want to install it.

• For non-RAID, choose the SATA/sSATA AHCI driver indicated then choose the storage drive on which you want to install it.

• Once all devices are specified, continue with the installation.

• After the Windows OS installation has completed, the system will automatically reboot multiple times.

5.2 Driver Installation

The Supermicro website contains drivers and utilities for your system at https://www.supermicro.com/wdl/driver. Some of these must be installed, such as the chipset driver.

After accessing the website, go into the CDR_Images (in the parent directory of the above link) and locate the ISO file for your motherboard. Download this file to a USB flash drive or a DVD. (You may also use a utility to extract the ISO file if preferred.)

Another option is to go to the Supermicro website at http://www.supermicro.com/products/. Find the product page for your motherboard, and "Download the Latest Drivers and Utilities". Insert the flash drive or disk and the screenshot shown below should appear.

SUPERMICRO X12DSC-6 Motherboard Drivers & Tools (Win2019) Intel® C621A Chipset X12DSC-6 SUPERMICRO Computer Inc. Intel Chipset INF files Microsoft .Net Framework 4.8 (Optional) ASPEED Graphics Driver Intel Virtual RAID on CPU Intel PRO Network Connections Drivers SUPERMICRO SuperDoctor 5 Build driver diskettes and manuals Browse CD Auto Start Up Next Time For more information, please visit SUPERMICRO's web site.

Figure 5-3. Driver and Tool Installation Screen

Note: Click the icons showing a hand writing on paper to view the readme files for each item. Click the computer icons to the right of these items to install each item (from top to the bottom) one at a time. After installing each item, you must re-boot the system before moving on to the next item on the list. The bottom icon with a CD on it allows you to view the entire contents.

5.3 SuperDoctor® 5

The Supermicro SuperDoctor 5 is a program that functions in a command-line or web-based interface for Windows and Linux operating systems. The program monitors such system health information as CPU temperature, system voltages, system power consumption, fan speed, and provides alerts via email or Simple Network Management Protocol (SNMP).

SuperDoctor 5 comes in local and remote management versions and can be used with Nagios to maximize your system monitoring needs. With SuperDoctor 5 Management Server (SSM Server), you can remotely control power on/off and reset chassis intrusion for multiple systems with SuperDoctor 5 Management Server monitors HTTP, FTP, and SMTP services to optimize the efficiency of your operation.

SuperDoctor® Manual and Resources

SuperDoctor 5 Mothearboard: C7S360-CB-ML Voltage 12V 24V 32V 40V 48V 56V 64V 72V 80V 88V 96V 104V 112V 120V 128V 136V 144V 152V 160V 168V 176V 184V 192V 200V 208V 216V 224V 232V 240V 248V 256V 264V 272V 280V 288V 296V 304V 312V 320V 328V 336V 344V 352V 360V 368V 376V 384V 392V 400V 408V 416V 424V 432V 440V 448V 456V 464V 472V 480V 488V 496V 504V 512V 520V 528V 536V 544V 552V 560V 568V 576V 584V 592V 600V 608V 616V 624V 632V 640V 648V 656V 664V 672V 680V 688V 696V 704V 712V 720V 728V 736V 744V 752V 760V 768V 776V 784V 792V 800V 808V 816V 824V 832V 840V 848V 856V 864V 872V 880V 888V 896V 904V 912V 920V 928V 936V 944V 952V 960V 968V 976V 984V 992V 1000V

Figure 5-4. SuperDoctor 5 Interface Display Screen (Health Information)

5.4 BMC

The X12DSC-6 supports the Baseboard Management Controller (BMC). The motherboard provides remote access, monitoring and management through the baseboard management controller (BMC) and other management controllers distributed among different system modules. There are several BIOS settings that are related to BMC. For general documentation and information on BMC, visit our website at: https://www.supermicro.com/en/solutions/management-software/bmc-resources.

BMC ADMIN User Password

For security, each system is assigned a unique default BMC password for the ADMIN user. This can be found on a sticker on the chassis and a sticker on the motherboard. The sticker also displays the BMC MAC address.

BMC AC1F6BC PWD SUOKJ

Figure 5-5. BMC Password Label

See Chapter 1 for label location.

Chapter 6

Optional Components

This chapter describes optional system components and installation procedures.

6.1 Optional Parts List

Optional Parts List
Type Part Number Description
Add-on Module 3616 (IT Mode) (per node)	AOM-S3616-S-O	Storage module with two 3616 controllers, PCI switch, and two NVMe M.2 connectors.
Add-on Card 3908 (HW RAID) (occupies PCIe slot1) and related parts as shown	AOC-S3908L-H8IR	AOC with SAS3908 controller (U3) and 240PD RAID Key (U39).
	AOM-SADPT-S Bypass card	with PCI switch.
	BTR-CVPM05	Broadcom 05-50039-00 CacheVault w/ 24" remote extender.
	CBL-SAST-1213-100 Slimline	x8 to Slimline x8,14-cm,100 OHM cable.

6.2 Intel Virtual RAID on CPU (VROC)

Intel® Virtual RAID on CPU (Intel VROC) is an enterprise RAID solution for NVMe SSDs directly attached to Intel Xeon Scalable processors. Intel Volume Management Device (VMD) is an integrated controller inside the CPU PCIe root complex.

• A single processor supports up to 12 NVMe SSDs and up to 6 RAID arrays.
• A dual processor system supports up to 24 NVMe SSDs and 12 RAID arrays.

Strip sizes are 4K, 8K, 16K, 32K, 64K, 128K.

Requirements and Restrictions

• Intel VROC is only available when the system is configured for UEFI boot mode.
• To enable the mdadm command and support for RSTe, install the patch from

- Linux: https://downloadcenter.intel.com/download/28158/Intel-Virtual-RAID-on-CPU-Intel-VROC-and-Intel-Rapid-Storage-Technology-enterprise-Intel-RSTe-Driver-for-Linux-

- Windows: https://downloadcenter.intel.com/download/28108/Intel-Virtual-RAID-on-CPU-Intel-VROC-and-Intel-Rapid-Storage-Technology-enterprise-Intel-RSTe-Driver-for-Windows-

• To enable Intel VROC, a hardware key must be inserted on the motherboard, and the appropriate processor's Virtual Management Devices must be enabled in the BIOS setup.
• It is possible to enable Intel VROC without a hardware key installed, but only RAID0 will be enabled.
• Intel VROC is not compatible with secure boot. This feature must be disabled.
• When creating bootable OS RAID1 devices, you must have both devices on the same CPU, and a VMD on that CPU.
• Spanning drives when creating RAID devices is not recommended to due to performance issues, even though it is supported.

Supported SSDs and Operating Systems

To see the latest support information: https://www.intel.com/content/www/us/en/support/articles/000030310/memory-and-storage/ssd-software.html

Additional Information

Additional information is available on the product page for the Supermicro add-on card and the linked manuals.

https://www.supermicro.com/en/products/accessories/addon/AOC-VROCxxxMOD.php

Hardware Key

The Intel VROC hardware key is a license key that detects the Intel VROC SKU and activates the function accordingly. The key must be plugged into the Supermicro motherboard (connector JRK1). The key options are:

Intel® VROC Keys
VROC Package Description Part Number Intel MM Number
Standard	RAID 0, 1, 10Supports 3rd party SSDs	AOC-VROCSTNMOD 951605
Premium	RAID 0, 1, 5, 10Supports 3rd party SSDs	AOC-VROCPREMOD 951606
Intel SSD only	RAID 0, 1, 5, 10Supports Intel SSDs only	AOC-VROCINTMOD 956822

VROC_HW_KEY PIN 1 GND KEY1 GND KEY2

Figure 6-1. Intel® VROC RAID Key and Motherboard Connector JRK1

Enabling NVMe RAID

RAID for NVMe SSDs must be enabled through the UEFI BIOS.

1. Install the patch as described in the Restrictions and Requirements section on a previous page.
2. Reboot the server.
3. Press [DEL] key to enter BIOS.
4. Switch to Advanced > Chipset Configuration > North Bridge > IIO Configuration > Intel® VMD Technology > CPU3 & CPU4.

5. Enable the VMD according to the following rules.

6. For U.2 NVMe, enable all the sub-items under each PStack, based on the your model server:

7. For M.2 NVMe or NVMe AIC, enable the VMD according to which AOC card/slot it used.

Examples for some U.2 configurations follow.

1. Press [F4] to save the configuration and reboot the system.
2. Press [DEL] to enter BIOS.
3. Switch to Advanced > Intel(R) Virtual RAID on CPU > All Intel VMD Controllers > Create RAID Volume.
4. Set Name.
5. Set RAID Level.
6. If cross-controller RAID is required, select Enable RAID spanned over VMD Controller as shown in Figure 6-4.
7. Select specific disks for RAID with an [X].

• RAID0: Select at least two [2 - 24] disks
• RAID1: Select only two disks
• RAID5: Select at least three [3 - 24] disks
• RAID10: Select only four disks

Optio Setup - PMI PID Configuration CPU10 Configuration CPU20 Configuration CPU30 Configuration CPU40 Configuration CSET Configuration InterM ST for Controlled I/O (IVT-d) USB Web Technologies PCI-E ASPM Support1 (RIsled) Digital Press: Enter to bring up the Intel XWD for Volume Management Device Configuration menu. ←: Select Screen F1: Select Item Enter: Select +/- Change Off. F1: General Help F2: Previous Values F3: Optimized Defaults F4: Save & Exit F5: Exit Version 6.11.17M Co#ovant 101 1201-1901

Advanced Intel30 VHD Technology • Intel30 VHD for Value Management Device on CPU • Intel30 VHD for Value Management Device on CPU • Intel30 VHD for Value Management Device on CPU +1: Select Screen Fid: Select Item Enter! Select +2: Change Out. Fid General Help Fid Previous Values Fid Optimized Defaults. Fid Save & Exit ESC: Exit version 6.21.1276 Copyright: HCI 2007-481

VMD Config for 10U 6 Enable/Disable VMD NameID: VMD NameID: VMD NameID: VMD Hot Plug Capable VMD Config for 10U 1 Enable/Disable VMD NameID: VMD NameID: VMD NameID: VMD Hot Plug Capable VMD Config for 10U 2 Enable/Disable VMD NameID: VMD NameID: VMD NameID: VMD Hot Plug Capable Enable/Disable VMD NameID: VMD NameID: VMD NameID: VMD Enable/Disable VMD NameID: VMD NameID: VMD Enable/Disable VMD Enable/Disable Net Plug for File Host Parts Enable/Disable Net Plug for File Host Parts Enable/Disable Net Plug for File Host Parts Enable/Disable Net Plug for File Host Parts Enable/Disable Net Plug for File Host Parts Enable/Disable Net Plug for File Host Parts Enable/Disable Net Plug for File Host Parts Enable/Disable Net Plug for File Host Parts Enable/Disable Net Plug for File Host Parts Enable/Disable Net Plug for File Host Parts Enable/Disable Net Plug for File Host Parts Enable/Disable Net Plug for File Host Parts Enable/Disable Net Plug for File Host Parts Enable/Disable Net Plug for File Host Parts Enable/Disable Net Plug for File Host Parts Enable/Disable Net Plug for File Host Parts Enable/Disable Net Plug for File Host Parts Enable/Disable Net Plug for File Host Parts Enable/Disable Net Plug for File Host Parts Enable/Disable Net Plug for File Host Parts

MHD Config for I/O 0 Enable/Disable VHD [Enable] MHD VHD [Enable] MHD VHD [Enable] MHD VHD [Enable] MHD VHD [Enable] Not Plug Capable [Enable] MHD Config for I/O 1 Enable/Disable VHD [Enable] MHD VHD [Enable] MHD VHD [Enable] MHD VHD [Enable] MHD VHD [Enable] Not Plug Capable [Enable] MHD Config for I/O 2 Enable/Disable VHD [Enable] MHD VHD [Enable] MHD VHD [Enable] MHD VHD [Enable] MHD VHD [Enable] Not Plug Capable [Enable] Enable/Disable VHD in this Class. ##1 Select Screen F4: Select Item Enter Select <=1 Change Not F1: General Help F2: Previous Values F3: Updated Defaults F4: Save & Exit F5: Exit Version 2.21.1278 Copyright: I/O 1920-AMZ

Dioio Setup - KMI Note: Advanced Event Logs SPML Security Boot Save & Exit Server HE Information FOI SATA Configuration FOI iSPN Configuration Network Configuration KMO Server Configuration KCS/PEI/MF Configuration Super TO Configuration Serial Port Somatic Redirection ACPE Settings Trustee Consulting HTTP Boot Configuration JOSET Configuration Intel(R) Ethernet Controller ATIA for 500MHz-T - DC-ESI-EF301CIF0 Intel(R) Ethernet Controller ATIA for 160MHz-T - DC-ESI-EF301CE/FP Intel(R) Ethernet Controller ATIA for 19 GB/SBIL CFP+ DC-ESI-EF301BC/PA Intel(R) Ethernet Controller ATIA for 18 Chipit CFP+ DC-ESI-EF301CUPB R.S. Authenticate Configuration Select IP Control EBIT/CPU Driver Health This format allows the user to manage Intel® Virtual RAD on CPU Select Screen PA: Select Item Enter Select +/- Change Rpt. FE: General RAD FE: Previous Values PA: Optimized Defaults PA: Save & Exit ECCY Exit

Intel(M) VMDC with AMD Technology 1.0.6.2008 Upgrade Key: Prime No RAID values on the system Intel VMDC Managed Controllers: + all Intel VMD Controllers Select to set here Information about the Intel VMD Controllers +1: Select Screen F1: Select Item Enter: Select +/- Change Rate. F1: General Hold F2: Previous Values F3: Optimized Defaults F4: Save & Exit ESC: Exit* Version 2.21.1270 Copyright: HCI - 2020 AMD

Figure 6-2. BIOS VMD Setting Examples

All Intel VMD Controller ■ Create RMSI volume Non-RMID Physical Disk: ■ MTENALISTATION-AWILIZABYIES SN:1725083E578, 10246.558E Port 110, Slot 20, CPU3, VMD, DCF 0100.0 ■ MTENALISTATION-AWILIZABYIES SN:1725101E577F, 10246.558G Port 111, Slot 21, CPU3, VMD, DCF 02100.0 This plus allow you to create a RMSI volume +: Select Screen TI: Select Item Enter: Select +/- Change Opt. F1: General Help F2: Previous Values F3: Default/Default F4: Save & Exit ESC: Exit Version 2.21.27B Copyright ©1 2020 AM

1. Select Strip Size (Default 64KB).
2. Select Create Volume.
3. If another RAID is needed, start again at step 6.
4. Press [F4] to save and reboot.

Routio Setup - AMI Advanced Create RAID Volume Name: RAMD Level: Enable RAID scanned over VMI Control/Volume: Select Blocks: MT30ALLITHTCH-AMRIZAVATION ON112928105780, 100A6.55GB Part 1:0 CPU4 VMOS MT30ALLITHTCH-AMRIZAVATION ON112928105780, 100A6.55GB Part 1:1 CPU4 VMOS Strip Size: Continuous (MM): Create Volume X - to Select Disk + Select Screen T4: Select 200x Order: Select v/-s Change Opt. F1: General Help F2: Premium Values F3: Optimized Defaults F4: Save & Exit USB: Exit

Figure 6-3. Created Volume without enabling RAID spanned over VMD Controller

Hello Setup - AMI Create RMS Volume Name: RMS Levels: Create RMS powered over RMS Controllers: Select Blocks: HTTP:ALLITATCH-WR1280RES ON:13291855780, 19248.5508 Part 1/4 CPU:WMS HTTP:ALLITATCH-WR1280RES ON:132918557F, 19248.5508 Part 1/4 CPU:WMS Strip Size: Capacity (GB): Create Volume N - to Select Osm +1: Select Screen T1: Select Item Enter: Select +/-: Change Opt. P1: General Info P2: Previous Values P3: Not listed Defaults Fat: Save & Exit Edit: Exit Version 0.01.12 W Copyright © 2020 AMI

Figure 6-4. Created Volume with enabling RAID spanned over VMD Controller

Status Indications

An LED indicator on the drive carrier shows the RAID status of the drive.

Drive Carrier Status LED Indicator
Status State (red)
Normal function Off
Locating 4 Hz blink
Fault Solid on
Rebuilding 1 Hz Blink

IBPI SFF 8489 Defined Status LED States

Hot Swap Drives

Intel VMD enables hot-plug and hot-unplug for NVMe SSDs, whether from Intel or other manufacturers. Under vSphere ESXi, several steps are necessary to avoid potential stability issues. See the information at link [1] below.

Hot-unplug

1. Prevent devices from being re-detected during rescan:

esxcli storage core claiming autoclaim --enabled=false

1. Unmount the VMFS volumes on the device. Check [2] for details.
2. Detach the device. Check [3] for details.
3. Physically remove the device.

Hot-plug

• Physically install the device.

ESXi will automatically discover NVMe SSDs, but a manual scan may be required in some cases.

Related Information Links

[1] https://kb.vmware.com/s/article/2151404
[2] https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.storage.doc/GUID-1B56EF97-F60E-4F21-82A7-8F2A7294604D.html
[3] https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.storage.doc/GUID-F2E75F67-740B-4406-9F0C-A2D99A698F2A.html

Chapter 7

Troubleshooting and Support

7.1 Information Resources

Website

A great deal of information is available onW the Supermicro website, supermicro.com.

SUPERMICRO TECHNOLOGY & OUR PLANET Supermicro Resource-Saving Systems: Reduce Cost and Environmental Impact

Figure 7-1. Supermicro Website

• Specifications for servers and other hardware are available by clicking the menu icon, then selecting the Products option.
• The Support option offers downloads (manuals, BIOS/BMC, drivers, etc.), FAQs, RMA, warranty, and other service extensions.

Direct Links for the SSG-640SP-DE1CR60 System

SSG-640SP-DE1CR60 specifications page

X12DSC-6 motherboard page for links to the Quick Reference Guide, User Manual, validated storage drives, etc.

Direct Links for General Support and Information

Frequently Asked Questions

Add-on card descriptions

TPM User Guide

General Memory Configuration Guide: X12

SuperDoctor5 Large Deployment Guide

Direct Links (continued)

For validated memory, see our Product Resources page

Product Matrices page for links to tables summarizing specs for systems, motherboards, power supplies, riser cards, add-on cards, etc.

Security Center for recent security notices

Supermicro Phone and Addresses

7.2 Baseboard Management Controller (BMC)

The system supports the Baseboard Management Controller (BMC). BMC is used to provide remote access, monitoring and management. There are several BIOS settings that are related to BMC.

For general documentation and information on BMC, please visit our website at: https://www.supermicro.com/en/solutions/management-software/bmc-resources.

System Storage USD Control Firmware Update Sensor Readings System Hardware Version 9.10.04 BE7A Hardware Build Time Oct/03/20 Rectifier Version 1.8.0 BOS Version BOS Build Time CPU0 Version FL:05:06 BMC Mac Address AC:1F:68:3D:22:DC LAN 1 Mac Address AC:1F:60:CA:22:BA LAN 2 Mac Address AC:1F:50:CA:22:BB LAN 1 Mac Address DC:4.7A:DE:5E:7E LAN 2 Mac Address DC:4.7A:DE:5E:7F LAN 1 Mac Address AC:1F:BB:CF:37:FA LAN 2 Mac Address AC:1F:BB:CF:37:F8 Host Server Host Name Server IP Address 172.31 SB 213 IPv6 Address 1 fc60-0-0C ac11:0ff fc63-3d43 IPv6 Address 2 IPv6 Address 3 IPv6 Address 4 IPv6 Address 5 IPv6 Address 6 Power Consumption Min Peak Average Usage Vax Peak X:Time(min), Y:Power Consumption (Watt) Remote Console Preview JAVA plug-in HTTPS reload image

Figure 7-2. BMC Sample

7.3 Troubleshooting Procedures

Use the following procedures to troubleshoot your system. If you have followed all of the procedures below and still need assistance, refer to the Technical Support Procedures or Returning Merchandise for Service section(s) in this chapter. Power down the system before changing any non hot-swap hardware components.

No Power

1. As you try to power up the system, note any beep codes. Refer to the next section for details on beep codes.
2. Check that the power LED on the motherboard is on.

PWRLED

Figure 7-3. Location of the MB Power LED

1. Make sure that the power connector is connected to your power supply.
2. Make sure that no short circuits exist between the motherboard and chassis.
3. Disconnect all cables from the motherboard, including those for the keyboard and mouse.
4. Remove all add-on cards.
5. Install a CPU, a heatsink, connect the internal speaker (if applicable), and the power LED to the motherboard. Make sure that the heatsink is fully seated.
6. Use the correct type of onboard CMOS battery as recommended by the manufacturer. Check to verify that it still supplies \~3VDC. If it does not, replace it with a new one. Warning: To avoid possible explosion, do not install the battery upside down.
7. Verify that all jumpers are set to their default positions.
8. Check that the power supplies' input voltage operate at 100-120v or 180-240v.
9. Turn the power switch on and off to test the system

No Video

1. If the power is on but you have no video, remove all the add-on cards and cables.
2. As you try to power up the system, note any beep codes. Refer to the next section for details on beep codes.

System Boot Failure

If the system does not display POST (Power-On-Self-Test) or does not respond after the power is turned on, check the following:

Turn on the system with only one DIMM module installed. If the system boots, check for bad DIMM modules or slots by following the Memory Errors Troubleshooting procedure below.

Memory Errors

1. Make sure that the DIMM modules are properly and fully installed.
2. Confirm that you are using the correct memory. Also, it is recommended that you use the same memory type and speed for all DIMMs in the system. See Section 3.5 for memory details.
3. Check for bad DIMM modules or slots by swapping modules between slots and noting the results.
4. Check the power supply voltage 115V/230V switch.

Losing the System's Setup Configuration

1. Make sure that you are using a high quality power supply. A poor quality power supply may cause the system to lose the CMOS setup information.
2. The battery on your motherboard may be old. Check to verify that it still supplies \~3VDC. If it does not, replace it with a new one.
3. If the above steps do not fix the setup configuration problem, contact your vendor for repairs.

When the System Becomes Unstable

If the system becomes unstable during or after OS installation, check the following:

1. CPU/BIOS support: Make sure that your CPU is supported and that you have the latest BIOS installed in your system.

2. Memory support: Make sure that the memory modules are supported by testing the modules using memtest86 or a similar utility.

Note: Refer to the product page on our website at http://www.supermicro.com for memory and CPU support and updates.

1. HDD support: Make sure that all hard disk drives (HDDs) work properly. Replace the bad HDDs with good ones.
2. System cooling: Check the system cooling to make sure that all heatsink fans and CPU/system fans, etc., work properly. Check the hardware monitoring settings in the BMC to make sure that the CPU and system temperatures are within the normal range. Also check the front panel Overheat LED and make sure that it is not on.
3. Adequate power supply: Make sure that the power supply provides adequate power to the system. Make sure that all power connectors are connected. Please refer to our website for more information on the minimum power requirements.
4. Proper software support: Make sure that the correct drivers are used.

If the system becomes unstable before or during OS installation, check the following:

1. Source of installation: Make sure that the devices used for installation are working properly, including boot devices such as CD.
2. Cable connection: Check to make sure that all cables are connected and working properly.
3. Using the minimum configuration for troubleshooting: Remove all unnecessary components (starting with add-on cards first), and use the minimum configuration (but with a CPU and a memory module installed) to identify the trouble areas. Refer to the steps listed in Section A above for proper troubleshooting procedures.
4. Identifying bad components by isolating them: If necessary, remove a component in question from the chassis, and test it in isolation to make sure that it works properly. Replace a bad component with a good one.
5. Check and change one component at a time instead of changing several items at the same time. This will help isolate and identify the problem.
6. To find out if a component is good, swap this component with a new one to see if the system will work properly. If so, then the old component is bad. You can also install the component in question in another system. If the new system works, the component is good and the old system has problems.

7.4 BIOS Error Beep (POST) Codes

During the POST (Power-On Self-Test) routines, which are performed each time the system is powered on, errors may occur.

Non-fatal errors are those which, in most cases, allow the system to continue the boot-up process. The error messages normally appear on the screen.

Fatal errors are those which will not allow the system to continue the boot-up procedure. If a fatal error occurs, you should consult with your system manufacturer for possible repairs.

These fatal errors are usually communicated through a series of audible beeps. The table below lists some common errors and their corresponding beep codes encountered by users.

BIOS Error Beep (POST) Codes
Beep Code Error	Message Description
1 short Refresh Circuits have been reset (Ready to power up)
5 short, 1 long Memory error No memory detected in system
5 long, 2 short Display memory read/write error Video adapter missing or with faulty memory
1 long continuous System OH System overheat condition

Additional BIOS POST Codes

The AMI BIOS supplies additional checkpoint codes, which are documented online at http://www.supermicro.com/support/manuals/ ("AMI BIOS POST Codes User's Guide").

When BIOS performs the Power On Self Test, it writes checkpoint codes to I/O port 0080h. If the computer cannot complete the boot process, a diagnostic card can be attached to the computer to read I/O port 0080h (Supermicro p/n AOC-LPC80-20).

For information on AMI updates, please refer to http://www.ami.com/products/.

7.5 Crash Dump Using BMC

In the event of a processor internal error (IERR) that crashes your system, you may want to provide information to support staff. You can download a crash dump of status information using BMC.

Check the BMC Error Log

1. Access the BMC web interface.
2. Click the Server Health tab, then Event Log to verify an IERR error.

Host Identification Server: 172 831 040.125 User: ADMIN (Administrator) System Server Health Configuration Remote Control Virtual Media Maintenance Miscellaneous Help Server Health Sensor Readings Event Log Event Log For more special event log settings, please click here. This page displays events from the system's event log. You can choose a category from the pull-down box to filter the events and also sort them by clicking on a column header. Clear Event Log Save Select an event log category: All Events Event ID Time Stamp Sensor Name Sensor Type Description 1 2017/10/19 15:38:37 Processor IERR - Assertion 2 2017/10/19 15:59:20 Processor IERR - Assertion

Figure 7-4. BMC Event Log

In the event of an IERR, the BMC executes a crash dump. You must download the crash dump and save it.

7.6 UEFI BIOS Recovery

Warning: Do not upgrade the BIOS unless your system has a BIOS-related issue. Flashing the wrong BIOS can cause irreparable damage to the system. In no event shall Supermicro be liable for direct, indirect, special, incidental, or consequential damages arising from a BIOS update. If you do update the BIOS, do not shut down or reset the system while the BIOS is updating to avoid possible boot failure.

Overview

The Unified Extensible Firmware Interface (UEFI) provides a software-based interface between the operating system and the platform firmware in the pre-boot environment. The UEFI specification supports an architecture-independent mechanism that will allow the UEFI OS loader stored in an add-on card to boot the system. The UEFI offers clean, hands-off management to a computer during system boot.

Recovering the UEFI BIOS Image

A UEFI BIOS flash chip consists of a recovery BIOS block and a main BIOS block (a main BIOS image). The recovery block contains critical BIOS codes, including memory detection and recovery codes for the user to flash a healthy BIOS image if the original main BIOS image is corrupted. When the system power is turned on, the recovery block codes execute first. Once this process is complete, the main BIOS code will continue with system initialization and the remaining POST (Power-On Self-Test) routines.

Note 1: Follow the BIOS recovery instructions below for BIOS recovery when the main BIOS block crashes.

Note 2: When the BIOS recovery block crashes, you will need to follow the procedures to make a Returned Merchandise Authorization (RMA) request. Also, you may use the Supermicro Update Manager (SUM) Out-of-Band (https://www.supermicro.com.tw/products/nfo/SMS_SUM.cfm) to reflash the BIOS.

Recovering the Main BIOS Block with a USB Device

This feature allows the user to recover the main BIOS image using a USB-attached device without additional utilities used. A USB flash device such as a USB Flash Drive, or a USB CD/DVD ROM/RW device can be used for this purpose. However, a USB Hard Disk drive cannot be used for BIOS recovery at this time.

The file system supported by the recovery block is FAT (including FAT12, FAT16, and FAT32) which is installed on a bootable or non-bootable USB-attached device. However, the BIOS might need several minutes to locate the SUPER.ROM file if the media size becomes too large due to the huge volumes of folders and files stored in the device.

To perform UEFI BIOS recovery using a USB-attached device, follow the instructions below.

1. Using a different machine, copy the "Super.ROM" binary image file into the Root "\" directory of a USB device or a writable CD/DVD.

Note 1: If you cannot locate the "Super.ROM" file in your drive disk, visit our website at www.supermicro.com to download the BIOS package. Extract the BIOS binary image into a USB flash device and rename it "Super.ROM" for the BIOS recovery use.

Note 2: Before recovering the main BIOS image, confirm that the "Super.ROM" binary image file you download is the same version or a close version meant for your motherboard.

1. Insert the USB device that contains the new BIOS image ("Super.ROM") into your USB drive and reset the system when the following screen appears.
2. After locating the healthy BIOS binary image, the system will enter the BIOS Recovery menu as shown below.

SUPERIOR BMC IP:10.132.161.13 PEI--Could Not Find Recovery Image... 87

Note: At this point, you may decide if you want to start the BIOS recovery. If you decide to proceed with BIOS recovery, follow the procedures below.

Option Setup Utility - Copyright (C) 2017 American Megatrends, Inc. Main Advanced Event Log (PVI) Recovery Security Boot Save a Exit Please select blocks you want to update: Boot Black Boot Block Update [Enabled] [Enabled] ▶ Proceed with flash update Get this option to reset WAVM to default values ++ Select Screen T4: Select Item Enter: Select +/- Change Opt. F1: General Help F2: Previous Values F3: Optimized Defaults F4: Save & Exit ESC: Exit Version 2.19.1766 Copyright (C) 2017 American Megatrends, Inc.

1. When the screen as shown above displays, use the arrow keys to select the item "Proceed with flash update" and press the key. You will see the BIOS recovery progress as shown in the screen below.

Note: Do not interrupt the BIOS flashing process until it has completed.

1. After the BIOS recovery process is complete, press any key to reboot the system.

2. Using a different system, extract the BIOS package into a USB flash drive.

Auto Setup Utility - Copyright (C) 2017 American Megatrends, Inc. Recovery WARNING! System firmware is being updated. Keyboard is locked. DO NOT TURN THE POWER OFF!!! Once firmware update is completed press any key to reboot the system Program new data Write new boot block... 178 +: Select Screen T4: Select Item Enter: Select +/-: Change Opt. F1: General Help F2: Previous Values F3: Optimized Defaults F4: Save a Exit DSC: Exit Version 2.19.1266. Copyright (C) 2017 American Megatrends, Inc.

1. Press continuously during system boot to enter the BIOS Setup utility. From the top of the tool bar, select Boot to enter the submenu. From the submenu list, select Boot

Aoio Setup Utility - Copyright (CI) 2017 American HegaTrends, Inc. Recovery WARNING: System firmware is being updated. Keyboard is locked. ON NOT TURN THE POWER OFF !!! Once firmware update is completed Press any key to reboot the system Flash update Flash update completed. Press any key to reset the system +: Select Screen +: Select Item enter: Select +/-: Change Opt. F1: General Help F2: Previous Values F3: Optimized Defaults F4: Save a Exit ESG: EXIT Version 2.15.1926. Copyright (CI) 2017 American HegaTrends, Inc.

Option #1 as shown below. Then, set Boot Option #1 to [UEFI AP:UEFI: Built-in EFI Shell]. Press to save the settings and exit the BIOS Setup utility.

1. When the UEFI Shell prompt appears, type fs# to change the device directory path. Go to the directory that contains the BIOS package you extracted earlier from Step 6. Enter flash.nsh BIOSname.### at the prompt to start the BIOS update process.

Auto Setup Utility - Copyright (C) 2017 American Megatrends, Inc. Main Advanced Event Loss JPMS Security Boot Save & Exit Boot Configuration Boot mode select [DUAL] LEGACY to EFI Support [Disabled] FIXED BOOT ORDER Priorities Boot Option #3 [DEF1 HP/DEF1: Bui...] Boot Option #2 [COM/DVD] Boot Option #3 [USB Hard Disk] Boot Option #4 [USB CD/DVI] Boot Option #5 [USB Key/SanDisk] Boot Option #6 [USB Flopy] Boot Option #7 [USB Lan] Boot Option #8 [Network: IBM QE SI...] Boot Option #9 [UEFI Hard Disk] Boot Option #10 [UEFI DD/DVI] Boot Option #11 [UEFI USB Hard Disk] Boot Option #12 [UEFI USB CO/DVI] Boot Option #13 [UEFI USB Key/UEFI...] Boot Option #14 [UEFI USB Flopy] Boot Option #15 [UEFI USB Lan] Boot Option #16 [UEFI Network] Boot Option #17 [Hard Disk] ▶ Hold New Boot Option Sets the system boot order +: Select Screen T4: Select Item Enter: Select +/-: Change Opt. F1: General help F2: Previous Values F3: Optimized Defaults F4: Save & Exit ESC: Exit Version 2.19.1265 Copyright (C) 2017 American Megatrends, Inc.

Note: Do not interrupt this process until the BIOS flashing is complete.

UIFI Interactive Shell v2.1 E3K 11 UIFI v2.50 (American Regatrends, 0x008900C) Mapping table FS0: FILE0(1):H0D0(1):BLK1; PsRoot(0x0)/PCI(0x14,0x0)/USB(0x11,0x0)/NOI,HDR,0xG7901D2,0xB06,0x1 OR3592) BLK0: File(s): PsRoot(0x0)/PCI(0x14,0x0)/USB(0x11,0x0) Press ESR to 1 seconds to skills startup.msh or any other key to continue. Shell: FSD FS0: VAPLDOS FS0:VAPLDOS> CD SAPMHE2_0362012 FS0:VAPLDOS\SAPMHE2_0362017\flash.msh X13P07.SX4

1. The screen above indicates that the BIOS update process is complete. When you see the screen above, unplug the AC power cable from the power supply, clear CMOS, and

Done. [ Access OXOS Port Ex ] Readout INDEX 0x51: 0x18 Done. ********************************************************************** * Program BIOS and ME (including RDT) regions... ********************************************************************** AVI Firmware Update Utility v5.09.01.1317 Copyright ©2017 American Registrends Inc. All Rights Reserved. OPUD = 50652 Reading Flash ....... done - ME Data Size checking . ok - PFS checksums ....... ok - Check BookLayout ....... ok, Erasing Boot Block ....... done Updating Boot Block ....... done Verifying Boot Block ....... done Froiling Main Block ....... 0x00152000 (00)

plug the AC power cable in the power supply again to power on the system.

1. Press continuously to enter the BIOS Setup utility.

Verifying HCB Block ....... done - Update success for FDR - Update success for SE. - Successful Update Recovery Loader to (PRx!! - Successful Update MFSB11- - Successful Update FTR11- - Successful Update NFx, [081 and 19021] - Successful Update FLOD and UTOX11 - ML entire image update success!! WAVRNDG : System must power-off to have the changes take effect! Moving FS01:AFUDD05/SKJPME2_03162017/40x64,efi -> FS01:AFUDD05/SKJPME2_03162017/f df.sac - [cA] Moving FS01:AFUDD05/SKJPME2_03162017/aufemfix4,efi -> FS01:AFUDD05/SKJPME2_0316201 ?afuemfix4,sac - [cA] ********************************************************************** * * Please ignore this 'shell cannot read from file - device error' * Warning message due to it does not impact flashing process. * ********************************************************************** baating ' Delete successful. FSb3v.

1. Press to load the default settings.
2. After loading the default settings, press to save the settings and exit the BIOS Setup utility.

7.7 CMOS Clear

JBT1 is used to clear CMOS, which will also clear any passwords. Instead of pins, this jumper consists of contact pads to prevent accidentally clearing the contents of CMOS.

To Clear CMOS

1. First power down the system completely.
2. Remove the cover of the chassis to access the motherboard.
3. Remove the onboard battery from the motherboard.
4. Short the CMOS pads with a metal object such as a small screwdriver for at least four seconds.
5. Remove the screwdriver or shorting device.
6. Replace the cover, reconnect the power cords and power on the system.

Notes: Clearing CMOS will also clear all passwords.

Do not use the PW_ON connector to clear CMOS.

JBT1 contact pads

7.8 Where to Get Replacement Components

If you need replacement parts for your system, to ensure the highest level of professional service and technical support, purchase exclusively from our Supermicro Authorized Distributors/System Integrators/Resellers. A list can be found at: http://www.supermicro.com. Click the "Where to Buy" tab.

7.9 Reporting an Issue

Technical Support Procedures

Before contacting Technical Support, please take the following steps. If your system was purchased through a distributor or reseller, please contact them for troubleshooting services. They have the best knowledge of your specific system configuration.

1. Please review the Troubleshooting Procedures in this manual and Frequently Asked Questions on our website before contacting Technical Support.
2. BIOS upgrades can be downloaded from our website. Note: Not all BIOS can be flashed depending on the modifications to the boot block code.

3. If you still cannot resolve the problem, include the following information when contacting us for technical support:

4. System, motherboard, and chassis model numbers and PCB revision number

5. BIOS release date/version (this can be seen on the initial display when your system first boots up)
6. System configuration

An example of a Technical Support form is posted on our website. Distributors: For immediate assistance, please have your account number ready when contacting our technical support department by email.

Returning Merchandise for Service

A receipt or copy of your invoice marked with the date of purchase is required before any warranty service will be rendered. You can obtain service by calling your vendor for a Returned Merchandise Authorization (RMA) number. When returning to the manufacturer, the RMA number should be prominently displayed on the outside of the shipping carton, and mailed prepaid or hand-carried. Shipping and handling charges will be applied for all orders that must be mailed when service is complete.

For faster service, RMA authorizations may be requested online (http://www.supermicro.com/support/rma/).

Whenever possible, repack the chassis in the original Supermicro carton, using the original packaging material. If these are no longer available, be sure to pack the chassis securely, using packaging material to surround the chassis so that it does not shift within the carton and become damaged during shipping.

This warranty only covers normal consumer use and does not cover damages incurred in shipping or from failure due to the alteration, misuse, abuse or improper maintenance of products.

During the warranty period, contact your distributor first for any product problems.

Vendor Support Filing System

For issues related to Intel, use the Intel IPS filing system:

https://www.intel.com/content/www/us/en/design/support/ips/training/welcome.html

For issues related to Red Hat Enterprise Linux, since it is a subscription based OS, contact your account representative.

7.10 Feedback

Supermicro values your feedback as we strive to improve our customer experience in all facets of our business. Please email us at [e-mail] to provide feedback on our manuals.

7.11 Contacting Supermicro

Headquarters

Address: Super Micro Computer, Inc.

980 Rock Ave.

San Jose, CA 95131 U.S.A.

Tel: [phone]

Fax: [phone]

Email: [e-mail] (General Information)

[e-mail] (Technical Support)

Website: www.supermicro.com

Europe

Address: Super Micro Computer B.V.

's-Hertogenbosch, The Netherlands

Tel: [phone]

Fax: [phone]

Email: [e-mail] (General Information)

[e-mail] (Technical Support)

[e-mail] (Customer Support)

Website: www.supermicro.nl

Asia-Pacific

Address: Super Micro Computer, Inc.

3F, No. 150, Jian 1st Rd.

Zhonghe Dist., New Taipei City 235

Taiwan (R.O.C)

Tel: [phone]

Fax: [phone]

Email: [e-mail]

Website: www.supermicro.com.tw

Appendix A

Standardized Warning Statements for AC Systems

About Standardized Warning Statements

The following statements are industry standard warnings, provided to warn the user of situations which have the potential for bodily injury. Should you have questions or experience difficulty, contact Supermicro's Technical Support department for assistance. Only certified technicians should attempt to install or configure components.

Read this appendix in its entirety before installing or configuring components in the Supermicro chassis.

These warnings may also be found on our website at http://www.supermicro.com/about/policies/safety_information.cfm.

Warning Definition

Warning! This warning symbol means danger. You are in a situation that could cause bodily injury. Before you work on any equipment, be aware of the hazards involved with electrical circuitry and be familiar with standard practices for preventing accidents.

警告の定義

この警告サインは危険を意味します。

Installation Instructions

Warning! Read the installation instructions before connecting the system to the power source.

設置手順書

Warning! This product relies on the building's installation for short-circuit (overcurrent) protection. Ensure that the protective device is rated not greater than: 250 V, 20 A.

サーキット・ブレーカー

Power Disconnection Warning

Warning! The system must be disconnected from all sources of power and the power cord removed from the power supply module(s) before accessing the chassis interior to install or remove system components.

CAUTION: This unit has redundant power sources. Please disconnect all the power cords before servicing.

電源切断の警告

Equipment Installation

Warning! Only trained and qualified personnel should be allowed to install, replace, or service this equipment.

機器の設置

Warning! This unit is intended for installation in restricted access areas. A restricted access area can be accessed only through the use of a special tool, lock and key, or other means of security. (This warning does not apply to workstations).

アクセス制限区域

Warning! There is the danger of explosion if the battery is replaced incorrectly. Replace the battery only with the same or equivalent type recommended by the manufacturer. Dispose of used batteries according to the manufacturer's instructions

電池の取り扱い

Redundant Power Supplies

Warning! This unit might have more than one power supply connection. All connections must be removed to de-energize the unit.

冗長電源装置

Warning! Hazardous voltage or energy is present on the backplane when the system is operating. Use caution when servicing.

バックプレーンの電圧

Comply with Local and National Electrical Codes

Warning! Installation of the equipment must comply with local and national electrical codes.

地方および国の電気規格に準拠

Warning! Ultimate disposal of this product should be handled according to all national laws and regulations.

製品の廃棄

Warning! Hazardous moving parts. Keep away from moving fan blades. The fans might still be turning when you remove the fan assembly from the chassis. Keep fingers, screwdrivers, and other objects away from the openings in the fan assembly's housing.

ファン・ホットスワップの警告

Power Cable and AC Adapter

Warning! When installing the product, use the provided or designated connection cables, power cables and AC adaptors. Using any other cables and adaptors could cause a malfunction or a fire. Electrical Appliance and Material Safety Law prohibits the use of UL or CSA-certified cables (that have UL/CSA shown on the cord) for any other electrical devices than products designated by Supermicro only.

電源コードとACアダプター

System Specifications

Processor Support

Dual 3rd Generation Intel® Xeon® Scalable processors in an LGA4189 socket; UPI up to 10.4GT/s; supports CPU TDP up to 205W Note: Refer to the motherboard specifications pages on our website for updates to supported processors.

Chipset

Intel® C621A

BIOS

AMI 128Mb SPI Flash EEPROM

Memory

4TB of 3DS LRDIMM/LRDIMM/3DS RDIMM/RDIMM DDR4 (288-pin) ECC memory with speeds of 3200/2933/2666 MHz in 16 memory slots and up to 4TB of Intel Optane PMem 200 Series with speeds of up to 3200 MHz

Storage Drives

60 x3.5/2.5" hot-swap SAS3/SATA3 drive bays (each node controls 30 drives) Two 7mm or one 15mm internal NVMe drive bays (optional) Two internal 2.5" SATA3 drive bays (per node) Two NVMe M.2 (per node)

PCI Expansion Slots

Three PCIe 4.0 x16 low-profile slots (CPU SLOT 1, 2, 3) Two PCIe 4.0 x2 NVMe M.2 slots (supports M-Key 2280 and 22110, on mezzanine board)

Input/Output

Network: Two 10GBaseT Ethernet LAN ports (Intel X550 controller) One RJ45 dedicated BMC LAN port One serial (COM) port One VGA port Two USB 3.0 ports (on the rear I/O panel)

Motherboard

X12DSC-6; Length 14.8", width 10.9" (376 mm x 277 mm)

Chassis

CSV-947STS-R2K63P; 4U Rackmount, 7 x 17.6 x 34.1 in. / 178 x 447 x 866mm

System Cooling

Six 8-cm heavy duty fans

Power Supply

Model: PWS-2K63A-1R, Titanium Level Power Supply (80 Plus) AC Input Voltages: 200-240 Vac Rated Input Current: 15A max. (200-240V) Rated Input Frequency: 50-60 Hz Rated Output Power: 2600W Rated Output Voltages: 200-240V: +12V (216 max.), +12Vsb (3.5A max)

Operating Environment

Operating Temperature: 10° to 35° C (50° to 95° F)

Non-operating Temperature: -40^ to 60^ C ( -40^ to 140^ F)

Operating Relative Humidity: 8% to 90% (non-condensing)

Non-operating Relative Humidity: 5% to 95% (non-condensing)

Regulatory Compliance

FCC, ICES, CE, UKCA, VCCI, RCM, NRTL, CB

Applied Directives, Standards

Perchlorate WarningEMC/EMI: 2014/30/EU (EMC Directive)

Electromagnetic Compatibility Regulations 2016

FCC Part 15

ICES-003

VCCI-CISPR 32

AS/NZS CISPR 32

BS/EN 55032

BS/EN 55035

CISPR 32

CISPR 24/CISPR 35

BS/EN 61000-3-2

BS/EN 61000-3-3

BS/EN 61000-4-2

BS/EN 61000-4-3

BS/EN 61000-4-4

BS/EN 61000-4-5

BS/EN 61000-4-6

BS/EN 61000-4-11

Product Safety: 2014/35/EU (LVD Directive)

UL/CSA 62368-1 (USA and Canada)

Electrical Equipment (Safety) Regulations 2016

IEC/BS/EN 62368-1

Environment:

2011/65/EU (RoHS Directive)

EC 1907/2006 (REACH)

2012/19/EU (WEEE Directive)

California Proposition 65

Warning! This product can expose you to chemicals including lead, known to the State of California to cause cancer and birth defects or other reproductive harm. For more information, go to www.P65Warnings.ca.gov.

Perchlorate Warning

California Best Management Practices Regulations for Perchlorate Materials: This Perchlorate warning applies only to products containing CR (Manganese Dioxide) Lithium coin cells. Perchlorate Material-special handling may apply. See www.dtsc.ca.gov/hazardouswaste/perchlorate