SuperServer 5039MD8-H8TNR - Server Supermicro - Free user manual and instructions

USER MANUAL SuperServer 5039MD8-H8TNR Supermicro

Front panel diagram of a server rack with labeled ports and control buttons

USER'S MANUAL

Revision 1.0

The information in this User's Manual has been carefully reviewed and is believed to be accurate. The vendor assumes no responsibility for any inaccuracies that may be contained in this document, and makes no commitment to update or to keep current the information in this manual, or to notify any person or organization of the updates. Please Note: For the most up-to-date version of this manual, please see our website at www.supermicro.com.

Super Micro Computer, Inc. ("Supermicro") reserves the right to make changes to the product described in this manual at any time and without notice. This product, including software and documentation, is the property of Supermicro and/or its licensors, and is supplied only under a license. Any use or reproduction of this product is not allowed, except as expressly permitted by the terms of said license.

IN NO EVENT WILL Super Micro Computer, Inc. BE LIABLE FOR DIRECT, INDIRECT, SPECIAL, INCIDENTAL, SPECULATIVE OR CONSEQUENTIAL DAMAGES ARISING FROM THE USE OR INABILITY TO USE THIS PRODUCT OR DOCUMENTATION, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. IN PARTICULAR, SUPER MICRO COMPUTER, INC. SHALL NOT HAVE LIABILITY FOR ANY HARDWARE, SOFTWARE, OR DATA STORED OR USED WITH THE PRODUCT, INCLUDING THE COSTS OF REPAIRING, REPLACING, INTEGRATING, INSTALLING OR RECOVERING SUCH HARDWARE, SOFTWARE, OR DATA.

Any disputes arising between manufacturer and customer shall be governed by the laws of Santa Clara County in the State of California, USA. The State of California, County of Santa Clara shall be the exclusive venue for the resolution of any such disputes. Supermicro's total liability for all claims will not exceed the price paid for the hardware product.

FCC Statement: This equipment has been tested and found to comply with the limits for a Class A digital device pursuant to Part 15 of the FCC Rules. These limits are designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial environment. This equipment generates, uses, and can radiate radio frequency energy and, if not installed and used in accordance with the manufacturer's instruction manual, may cause harmful interference with radio communications. Operation of this equipment in a residential area is likely to cause harmful interference, in which case you will be required to correct the interference at your own expense.

California Best Management Practices Regulations for Perchlorate Materials: This Perchlorate warning applies only to products containing CR (Manganese Dioxide) Lithium coin cells. "Perchlorate Material-special handling may apply. See www.dtsc.ca.gov/hazardouswaste/perchlorate".

WARNING: This product can expose you to chemicals including lead, known to the State of California to cause cancer and birth defects or other reproductive harm. For more information, go to www.P65Warnings.ca.gov.

The products sold by Supermicro are not intended for and will not be used in life support systems, medical equipment, nuclear facilities or systems, aircraft, aircraft devices, aircraft/emergency communication devices or other critical systems whose failure to perform be reasonably expected to result in significant injury or loss of life or catastrophic property damage. Accordingly, Supermicro disclaims any and all liability, and should buyer use or sell such products for use in such ultra-hazardous applications, it does so entirely at its own risk. Furthermore, buyer agrees to fully indemnify, defend and hold Supermicro harmless for and against any and all claims, demands, actions, litigation, and proceedings of any kind arising out of or related to such ultra-hazardous use or sale.

Manual Revision 1.0

Release Date: March 27, 2019

mk

Unless you request and receive written permission from Super Micro Computer, Inc., you may not copy any part of this document. Information in this document is subject to change without notice. Other products and companies referred to herein are trademarks or registered trademarks of their respective companies or mark holders.

Copyright © 2019 by Super Micro Computer, Inc.

All rights reserved.

Printed in the United States of America

Preface

About this Manual

This manual is written for professional system integrators and PC technicians. It provides information for the installation and use of the server. Installation and maintenance should be performed by experienced technicians only.

Please refer to the 5039MD(8/18)-H8TNR server specifications page on our website for updates on supported memory, processors and operating systems (http://www.supermicro.com).

Notes

For your system to work properly, please follow the links below to download all necessary drivers/utilities and the user's manual for your server.

• Supermicro product manuals: http://www.supermicro.com/support/manuals/
- Product drivers and utilities: https://www.supermicro.com/wftp
- Product safety info: http://www.supermicro.com/about/policies/safety_information.cfm

If you have any questions, please contact our support team at:

[e-mail]

This manual may be periodically updated without notice. Please check the Supermicro website for possible updates to the manual revision level.

Warnings

Special attention should be given to the following symbols used in this manual.

Warning! Indicates important information given to prevent equipment/property damage or personal injury.

Warning! Indicates high voltage may be encountered when performing a procedure.

Contents

Chapter 1 Introduction

1.1 Overview....8
1.2 Unpacking the System 8
1.3 System Features 9
1.4 Chassis Features ....10

Control Panel 10
Chassis Front....11
Chassis Rear....12

1.5 Motherboard Layout....13

Quick Reference 14
System Block Diagram....15

1.6 Where to Get Replacement Components....16
1.7 Returning Merchandise for Service....16

Chapter 2 Server Installation

2.1 Overview....17
2.2 Preparing for Setup....17

Choosing a Setup Location....17
Rack Precautions....17
Server Precautions....18
Rack Mounting Considerations....18

Ambient Operating Temperature....18
Airflow....18
Mechanical Loading....18
Circuit Overloading....19
Reliable Ground....19

2.3 Installing the Rails....20

Identifying the Rails....20
Releasing the Inner Rail 21
Installing the Inner Rails on the Chassis....22
Installing the Outer Rails onto the Rack....23

2.4 Installing the Chassis into a Rack....24

Removing the Chassis from the Rack....25

Chapter 3 Maintenance and Component Installation

3.1 Removing Power....26
3.2 Accessing the System....27

Removing a Computing Node ....27

Removing the Chassis Cover ....28

3.3 Motherboard Components....29

Memory 29

Memory Support....29

DIMM Module Population Configuration....29

Memory Population Guidelines....30

Memory Population Sequence ....31

Installing Memory....32

Motherboard Battery ....33

3.4 Chassis Components ....34

Corresponding Nodes, Fans and Hard Drives....34

Storage Drives 35

Drive Carrier Indicators....35

M.2 Solid State Drives ....38

System Cooling 39

Fans....39

Installing the Air Shroud ....40

Power Supply 41

Power Supply LEDs....41

PCI Expansion Cards ....43

Chapter 4 Motherboard Connections

4.1 Power Connections ....45

4.2 Headers and Connectors ....45

4.3 Input/Output Ports 47

Rear I/O Ports 47

LAN Ports 47

4.4 Jumpers....48

Explanation of Jumpers....48

4.5 LED Indicators....50

Chapter 5 Software

5.1 Microsoft Windows OS Installation....51
5.2 Driver Installation....53
5.3 SuperDoctor ^® 5....54
5.4 IPMI 55

Chapter 6 BIOS

6.1 Introduction....56
Starting BIOS Setup Utility....56
6.2 Main Setup 56
6.3 Advanced Setup Configurations....58
6.4 IPMI 85
6.5 Security....88
6.6 Event Logs 92
6.7 Boot....93
6.8 Save & Exit....95

Appendix A BIOS Error Codes

Appendix B Standardized Warning Statements for AC Systems

Appendix C System Specifications

Appendix D UEFI BIOS Recovery

Appendix E IPMI Crash Dump

Appendix F CPU-Based RAID for NVMe

Contacting Supermicro

Headquarters

Address: Super Micro Computer, Inc.

980 Rock Ave.

San Jose, CA 95131 U.S.A.

Tel: [phone]

Fax: [phone]

Email: [e-mail] (General Information)

[e-mail] (Technical Support)

Website: www.supermicro.com

Europe

Address: Super Micro Computer B.V.

's-Hertogenbosch, The Netherlands

Tel: [phone]

Fax: [phone]

Email: [e-mail] (General Information)

[e-mail] (Technical Support)

[e-mail] (Customer Support)

Website: www.supermicro.nl

Asia-Pacific

Address: Super Micro Computer, Inc.

3F, No. 150, Jian 1st Rd.

Zhonghe Dist., New Taipei City 235

Taiwan (R.O.C)

Tel: [phone]

Fax: [phone]

Email: [e-mail]

Website: www.supermicro.com.tw

Chapter 1

Introduction

1.1 Overview

This chapter provides a brief outline of the functions and features of the 5039MD(8/18)-H8TNR server. It is a 3U Micro-cloud system with eight computing nodes, each with an X11SDD-(8/18)C-F motherboard, all enclosed in the SC938NH-R1K68B chassis.

In addition to the motherboards and chassis, several important parts that are included with the system are listed below.

Main Parts List
Description Part Number Quantity
Power supply modules PWS-1K68A-1R 2
Storage drive backplane AOM-BPN-938NH-P 1
Power distributors	AOM-PDB-PT938-LSGAOM-PDB-PT938-RSG	1 each
Network cards AOC-CGP-I2-P 8
Add-on cards AOM-LAN-MC8-P 1
Fans FAN-0194L4 4
Air shrouds MCP-310-93805-0B 16
Riser cards RSC-RR1U-E16 8
Rack mount rails	MCP-290-00057-0N	1 set

1.2 Unpacking the System

Inspect the box the system was shipped in and note if it was damaged in any way. If any equipment appears damaged, please file a damage claim with the carrier who delivered it.

Decide on a suitable location for the rack unit that will hold the server. It should be situated in a clean, dust-free area that is well ventilated. Avoid areas where heat, electrical noise and electromagnetic fields are generated. It will also require a grounded AC power outlet nearby. Be sure to read the precautions and considerations noted in Appendix B.

1.3 System Features

The following table is an overview of the main features of the 5039MD(8/18)-H8TNR server.

System Features

Motherboard (per node)

X11SDD-8C-F for 5039MD8-H8TNR; X11SDD-18C-F for 5039MD18-H8TNR

Chassis

SC938NH-R1K68B

CPU (per node)

For 5039MD8-H8TNR single Intel Xeon D-2141I processor, 8 Core, 16 Threads, 65W, System on Chip For 5039MD18-H8TNR single Intel Xeon D-2191 processor, 18 Core, 36 Threads, 86W, System on Chip

Memory (per node)

Up to 512GB ECC LRDIMM or 256GB ECC RDIMM in four DDR4 slots with speed of up to 2400MHz

Expansion Slots (per node)

One low-profile PCI-E x16 slot; one micro-LP

Storage Drives

Sixteen (two per node) 3.5" SATA3 by default; optional kits allow sixteen 2.5" hybrid SATA3/NVMe M.2: Up to two PCI-E M.2 SSDs (per node)

Power

Dual 1600 W redundant 80Plus Titanium level modules

Cooling

Four 8-cm heavy duty fans, CPU heatsinks, air shrouds to direct air flow

Input/Output (per node)

LAN: Two Gb Ethernet ports via Micro-LP network card With KVM dongle: two USB 2.0 ports, one VGA port; one COM port (per system) 1+1 dedicated IPMI LAN port for 8 nodes management

Form Factor

3U rackmount; (WxHxD) 17.26 x 5.21 x 23.2 in. (438 x 132 x 589 mm)

1.4 Chassis Features

Control Panel

Power switches and status LEDs are located on the control panel on the front of the chassis.

Control Panel Features
Item Features Description
1 Power button		The main power switch applies or removes primary power from the power supply to the chassis but maintains standby power.
2 Power Fail Indicates a failed power supply module
3 Node activity Indicates activity on the numbered node

Node Activity LED Indicators

LED Appearance Description

Solid Green The node is powered on and operating normally

Blinking Green The node is in the process of shutting down

Solid Red The node is detecting an overheated condition

1Hz Blinking Red The node is detecting a fan failure

.25Hz Blinking Red The node is detecting a power failure

Solid Blue The node local UID is on

1Hz Blinking Blue The node remote UID is on

No Illumination The node is powered-down

Chassis Front

The illustration below shows the features included on the front of the chassis.

Front panel diagram of a server rack with labeled ports and status indicators

Figure 1-2. Front View

Chassis Front Features
Item Features Description
1 Control	Panel Power buttons and status indicators; details on the previous page
2 Storage	Drives	Sixteen 3.5" drive bays, two for each computing node; drive carriers display status lights

Drive Carrier LED Indicators
	Color Blinking Pattern Behavior		for Device
Activity LED	Blue Solid On SAS/NVMe drive installed
	Blue Blinking I/O activity
Status LED	Red Solid On Failure of drive with RSTe support
	Red Blinking at 1 Hz Rebuild drive with RSTe support
	Red Blinking with two blinks and one stop at 1 Hz	Hot spare for drive with RSTe support (not supported in VMD mode)
	Red On for five seconds, then off	Power on for drive with RSTe support
	Red Blinking at 4 Hz Identify drive with RSTe support
	Green Solid On Safe to remove NVMe device (not supported in VMD mode)
	Amber Blinking at 1 Hz Attention state—do not remove NVMe device (not supported in VMD mode)

Chassis Rear

The illustration below shows the features included on the rear of the chassis. Computing nodes and power supply modules display status lights.

10 9 8 7 6 5 4 3 2 1 9

11 12 13 14

Figure 1-3. Rear View

Chassis Rear Features
Item Features Description
1 – 8 Nodes		Each node features a power button, two LAN ports, one dedicated IPMI LAN port, and a VGA port
9 Power	Supplies Two redundant power supply modules, PWS1 on the left, PWS2 on the right
10 IPMI LAN ports		Two ports: can be used in two ways: Cascade (connect the first port to a management device and the second port to another server) or Redundancy (connect each port to a different subnetwork switch).
11 UID Button Toggle switch		and LED indicator for Unit ID (blue)
12 Power	Button Node power switch and indicator (green)
13 KVM Port One VGA, one		COM and two USB 2.0 (with KVM dongle)
14 Fail LED		Indicator for node or corresponding storage drive failure (red); Overheat (OH); PWR Fail (.25HZ); Fan Fail LED (1HZ)

Close-up of electronic device ports including a USB connector and a circular connector (no visible text or symbols)

Figure 1-4. Node I/O Features

Connecting to a Node

USB, COM, and VGA port capabilities can be added to any node through the KVM port on the rear. Plug in the adapter (dongle).

Three black and blue cables with connectors, placed on a plain surface (no text or symbols visible)

Figure 1-5. Adapter to Provide USB, COM, and VGA Ports

1.5 Motherboard Layout

Below is a layout of the X11SDD-(8/18)C-F motherboard with jumper, connector and LED locations shown. See the table on the following page for descriptions. For detailed descriptions, pinout information and jumper settings, refer to Chapter 4.

SW1 J3 - M.2 JID LED2 J3 M.2-H J2 M.2-H J2 - M.2 JKVM1 USB0/1 LED3 MICRO-LP JRK1 JTPM1 JWD1 JPME2 BT1 JBT1 DIMMD1 DIMME1 FAN1 FAN1 CPU SRW10 SRW6 BMC HB LED1 SRW11 SRW7 SRW12 SRW8 SRW9 SRW5 PCI-E SLOT SRW6 SRW10 SRW7 SRW11 SRW8 SRW12 SRW9 SRW5 DIMMA1 DIMMB1 SUPER X11SDD-8C-F REV.1.02 DESIGNED IN USA PE CE S-SGPIO1 SAS0 SAS1 MH3 FAN1 MH6 MH1 JPWR1 JPWR1

Figure 1-6. Motherboard Layout

Quick Reference

Jumper Description Default Setting

JBT1 CMOS Clear Open (Normal) (See Chapter 4)

JPME2 Manufacture Mode Select (ME Mode Select) Pins 1-2 (Normal)

JWD1 Watch Dog Timer Enable Pins 1-2 (Reset to System)

Connector Description

BT1 Onboard Battery
FAN1 System Fan Header
J2 M.2 M-key Slot
J3 M.2 M-key Slot
JKVM1 Keyboard/Video/Mouse Connector
JPWR1 12V DC Power Connector
JRK1 Intel RAID Key Header
JTPM1 Trusted Platform Module (TPM) Header
MICRO-LP	CPU MICRO-LP PCI-E 3.0 x8 Slot
PCI-E Slot	PCI-E 3.0 x16 Slot
SAS0 - SAS1	Serial Attached SCSI 3.0 Ports when a SAS Add on Card is installed
S-SGPIO1	Serial General Purpose I/O Header
SRW5 - SRW12	M.2 Holding Screws
SW1	Power Switch
UID	Unit ID Switch
USB0/1	Back Panel USB 2.0 Ports

LED	Description	State: Status
BMC_HB_LED1	BMC Hearbeat LED	Blinking Green: BMC Normal
LED2	UID LED	Solid Blue: Unit Identified
LED3	System Failure LED	Solid Red: OverheatingBlinking Red: Power or Fan Failure

System Block Diagram

graph TD
    A["2 HDD PORTS NVMe/SAS/SATA (system front)"] --> B["MUX"]
    B --> C["2 NVMe PORTS"]
    C --> D["CPU"]
    D --> E["2 SASTA PORTS"]
    E --> F["PCH"]
    F --> G["BMC AST2500"]
    G --> H["RTL8211 PHY"]
    G --> I["VGA/COM (KVM)"]
    B --> J["2 USB PORTS(KVM)"]
    B --> K["2 M.2 PORTS (PCIE or SATA)"]
    B --> L["TPM (header)"]
    B --> M["FLASH SPI 256Mb"]
    B --> N["J25 jumper: 1-2: sSATA 2-3: SAS"]
    B --> O["AOM-BPN-938NH"]
    D --> P["DDR4 2566/2400/2133/1866 MHz"]
    D --> Q["DDR4 2566/2400/2133/1866 MHz"]
    D --> R["USB2.0 X 1"]
    D --> S["PCIe3.0 x 1"]
    D --> T["eSPI"]
    D --> U["PCIe3.0 x 4/SATA III"]
    D --> V["SATA III"]
    D --> W["SATA III"]
    D --> X["USB2.0 X 2"]
    D --> Y["PCIe3.0 x 4/SATA III"]
    D --> Z["PCIe3.0 x 4/SATA III"]
    D --> AA["SPI"]
    D --> AB["SPI"]
    D --> AC["PCIe3.0 x 4/SATA III"]
    D --> AD["PCIe3.0 x 4/SATA III"]
    D --> AE["PCIe3.0 x 4/SATA III"]
    D --> AF["PCIe3.0 x 4/SATA III"]
    D --> AG["PCIe3.0 x 4/SATA III"]
    D --> AH["PCIe3.0 x 4/SATA III"]
    D --> AI["PCIe3.0 x 4/SATA III"]
    D --> AJ["PCIe3.0 x 4/SATA III"]
    D --> AK["PCIe3.0 x 4/SATA III"]
    D --> AL["PCIe3.0 x 4/SATA III"]
    D --> AM["PCIe3.0 x 4/SATA III"]
    D --> AN["PCIe3.0 x 4/SATA III"]
    D --> AO["PCIe3.0 x 4/SATA III"]
    D --> AP["PCIe3.0 x 4/SATA III"]
    D --> AQ["PCIe3.0 x 4/SATA III"]
    D --> AR["PCIe3.0 x 4/SATA III"]
    D --> AS["PCIe3.0 x 4/SATA III"]
    D --> AT["PCIe3.0 x 4/SATA III"]
    D --> AU["PCIe3.0 x 4/SATA III"]
    D --> AV["PCIe3.0 x 4/SATA III"]
    D --> AW["PCIe3.0 x 4/SATA III"]
    D --> AX["PCIe3.0 x 4/SATA III"]
    D --> AY["PCIe3.0 x 4/SATA III"]
    D --> AZ["PCIe3.0 x 4/SATA III"]
    D --> BA["PCIe3.0 x 4/SATA III"]
    D --> BB["PCIe3.0 x 4/SATA III"]
    D --> BC["PCIe3.0 x 4/SATA III"]
    D --> BD["PCIe3.0 x 4/SATA III"]
    D --> BE["PCIe3.0 x 4/SATA III"]
    D --> BF["PCIe3.0 x 4/SATA III"]
    D --> BG["PCIe3.0 x 4/SATA III"]
    D --> BH["PCIe3.0 x 4/SATA III"]
    D --> BI["PCIe3.0 x 4/SATA III"]
    D --> BJ["PCIe3.0 x 4/SATA III"]
    D --> BK["PCIe3.0 x 4/SATA III"]
    D --> BL["PCIe3.0 x 4/SATA III"]
    D --> BM["PCIe3.0 x 4/SATA III"]
    D --> BN["PCIe3.0 x 4/SATA III"]
    D --> BO["PCIe3.0 x 4/SATA III"]
    D --> BP["PCIe3.0 x 4/SATA III"]
    D --> BQ["PCIe3.0 x 4/SATA III"]

Figure 1-7. System Block Diagram

Note: This is a general block diagram and may not exactly represent the features on your motherboard. See the System Specifications appendix for the actual specifications of your motherboard.

1.6 Where to Get Replacement Components

If you need replacement parts for your system, to ensure the highest level of professional service and technical support, purchase exclusively from our Supermicro Authorized Distributors/System Integrators/Resellers. A list can be found at: http://www.supermicro.com. Click the "Where to Buy" link.

1.7 Returning Merchandise for Service

A receipt or copy of your invoice marked with the date of purchase is required before any warranty service will be rendered. You can obtain service by calling your vendor for a Returned Merchandise Authorization (RMA) number. When returning to the manufacturer, the RMA number should be prominently displayed on the outside of the shipping carton, and mailed prepaid or hand-carried. Shipping and handling charges will be applied for all orders that must be mailed when service is complete.

For faster service, RMA authorizations may be requested online (http://www.supermicro.com/support/rma/).

Whenever possible, repack the chassis in the original Supermicro carton, using the original packaging material. If these are no longer available, be sure to pack the chassis securely, using packaging material to surround the chassis so that it does not shift within the carton and become damaged during shipping.

This warranty only covers normal consumer use and does not cover damages incurred in shipping or from failure due to the alteration, misuse, abuse or improper maintenance of products.

During the warranty period, contact your distributor first for any product problems.

Chapter 2

Server Installation

2.1 Overview

This chapter provides advice and instructions for mounting your system in a server rack. If your system is not already fully integrated with processors, system memory etc., refer to Chapter 4 for details on installing those specific components.

Caution: Electrostatic Discharge (ESD) can damage electronic components. To prevent such damage to PCBs (printed circuit boards), it is important to use a grounded wrist strap, handle all PCBs by their edges and keep them in anti-static bags when not in use.

2.2 Preparing for Setup

The box in which the system was shipped should include the rackmount hardware needed to install it into the rack. Please read this section in its entirety before you begin the installation.

Choosing a Setup Location

• The system should be situated in a clean, dust-free area that is well ventilated. Avoid areas where heat, electrical noise and electromagnetic fields are generated.
• Leave enough clearance in front of the rack so that you can open the front door completely (\~25 inches) and approximately 30 inches of clearance in the back of the rack to allow sufficient space for airflow and access when servicing.
• This product should be installed only in a Restricted Access Location (dedicated equipment rooms, service closets, etc.).
• This product is not suitable for use with visual display workplace devices according to §2 of the German Ordinance for Work with Visual Display Units.

Rack Precautions

- Ensure that the leveling jacks on the bottom of the rack are extended to the floor so that the full weight of the rack rests on them.

• In single rack installations, stabilizers should be attached to the rack. In multiple rack installations, the racks should be coupled together.
• Always make sure the rack is stable before extending a server or other component from the rack.
• You should extend only one server or component at a time - extending two or more simultaneously may cause the rack to become unstable.

Server Precautions

• Review the electrical and general safety precautions in Appendix B.
• Determine the placement of each component in the rack before you install the rails.
• Install the heaviest server components at the bottom of the rack first and then work your way up.
• Use a regulating uninterruptible power supply (UPS) to protect the server from power surges and voltage spikes and to keep your system operating in case of a power failure.
• Allow any drives and power supply modules to cool before touching them.
• When not servicing, always keep the front door of the rack and all covers/panels on the servers closed to maintain proper cooling.

Rack Mounting Considerations

Ambient Operating Temperature

If installed in a closed or multi-unit rack assembly, the ambient operating temperature of the rack environment may be greater than the room's ambient temperature. Therefore, consideration should be given to installing the equipment in an environment compatible with the manufacturer's maximum rated ambient temperature (TMRA).

Airflow

Equipment should be mounted into a rack so that the amount of airflow required for safe operation is not compromised.

Mechanical Loading

Equipment should be mounted into a rack so that a hazardous condition does not arise due to uneven mechanical loading.

Circuit Overloading

Consideration should be given to the connection of the equipment to the power supply circuitry and the effect that any possible overloading of circuits might have on overcurrent protection and power supply wiring. Appropriate consideration of equipment nameplate ratings should be used when addressing this concern.

Reliable Ground

A reliable ground must be maintained at all times. To ensure this, the rack itself should be grounded. Particular attention should be given to power supply connections other than the direct connections to the branch circuit (i.e. the use of power strips, etc.).

To prevent bodily injury when mounting or servicing this unit in a rack, you must take special precautions to ensure that the system remains stable. The following guidelines are provided to ensure your safety:

• This unit should be mounted at the bottom of the rack if it is the only unit in the rack.
• When mounting this unit in a partially filled rack, load the rack from the bottom to the top with the heaviest component at the bottom of the rack.
• If the rack is provided with stabilizing devices, install the stabilizers before mounting or servicing the unit in the rack.
• Slide rail mounted equipment is not to be used as a shelf or a work space.

2.3 Installing the Rails

There are a variety of rack units on the market, which may require a slightly different assembly procedure. This rail set fits a rack between 26.8" and 36.4" deep.

The following is a basic guideline for installing the system into a rack with the rack mounting hardware provided. You should also refer to the installation instructions that came with the specific rack you are using.

Identifying the Rails

The chassis package includes two rail assemblies. Each assembly consists of three sections: An inner rail that secures directly to the chassis, an outer rail that secures to the rack, and a middle rail which extends from the outer rail. These assemblies are specifically designed for the left and right side of the chassis and labeled.

Outer Rail Middle Rail Locking Tab Inner Rail

Figure 2-1. Identifying the Outer Rail, Middle Rail and Inner Rail (Left Rail Assembly Shown)

Releasing the Inner Rail

Each inner rail has a locking latch. This latch prevents the server from coming completely out of the rack when when the chassis is pulled out for servicing.

To mount the rail onto the chassis, first release the inner rail from the outer rails.

1. Pull the inner rail out of the outer rail until it is fully extended as illustrated below.
2. Press the locking tab down to release the inner rail.
3. Pull the inner rail all the way out.

Outer Rail 1 Middle Rail Locking Tab Inner Rail 2 3

Figure 2-2. Extending and Releasing the Inner Rail

Installing the Inner Rails on the Chassis

Installing the Inner Rails

1. Identify the left and right inner rails. They are labeled.
2. Place the inner rail firmly against the side of the chassis, aligning the hooks on the side of the chassis with the holes in the inner rail.
3. Slide the inner rail forward toward the front of the chassis until the quick release bracket snaps into place, securing the rail to the chassis.
4. Optionally, you can further secure the inner rail to the chassis with screws.

Inner Rails 4 2 3 4

Figure 2-3. Inner Rails Installed on the Chassis

Installing the Outer Rails onto the Rack

Each end of the assembled outer rail includes a bracket with hooks and square, spring-loaded pegs to fit into the square holes in your rack.

Installing the Outer Rail

1. Press upward on the locking tab at the rear end of the middle rail.
2. Push the middle rail back into the outer rail.
3. Hang the hooks on the front of the outer rail onto the square holes on the front of the rack. If desired, use screws to secure the outer rails to the rack.
4. Pull out the rear of the outer rail, adjusting the length until it just fits within the posts of the rack.
5. Hang the hooks of the rear section of the outer rail onto the square holes on the rear of the rack. Take care that the proper holes are used so the rails are level. If desired, use screws to secure the rear of the outer rail to the rear of the rack.

Technical diagram illustrating four stages of a mechanical assembly: linear tracking, vertical rail, horizontal rail, and vertical support.

Figure 2-4. Extending and Mounting the Outer Rails

Note: The figure above is for illustrative purposes only. Always install servers at the bottom of the rack first.

Stability hazard. The rack stabilizing mechanism must be in place, or the rack must be bolted to the floor before you slide the unit out for servicing. Failure to stabilize the rack can cause the rack to tip over.

Warning: Do not pick up the server with the front handles. They are designed to pull the system from a rack only.

2.4 Installing the Chassis into a Rack

Once rails are attached to the chassis and the rack, you can install the server.

Warning: Mounting the system into the rack requires at least two people to support the chassis during installation. Please follow safety recommendations printed on the rails.

Installing the Chassis into a Rack

1. Extend the outer rails.
2. Align the inner rails of the chassis with the outer rails on the rack.
3. Slide the inner rails into the outer rails, keeping the pressure even on both sides. When the chassis has been pushed completely into the rack, it should click into the locked position.
4. Optional screws may be used to hold the front of the chassis to the rack.

Technical line drawing of a server rack cabinet with rack-mounted panels and ventilation grilles (no text or labels)

Figure 2-5. Installing the Server into the Rack

Note: Keep the ball bearing shuttle locked at the front of the middle rail during installation.

Note: Figure is for illustrative purposes only. Always install servers to the bottom of a rack first.

Removing the Chassis from the Rack

Caution! The system is heavy. It is dangerous for a single person to remove it from the rack. Have sufficient personnel or use a lift to support the chassis.

1. Pull the chassis forward out the front of the rack until it stops.
2. Press the release latches on each of the inner rails downward simultaneously and continue to pull the chassis forward and out of the rack.

Technical line drawing of a server rack unit with ventilation slots and drive bays (no text or labels)

Figure 2-6. Removing the Chassis From the Rack

Chapter 3

Maintenance and Component Installation

This chapter provides instructions on installing and replacing main system components. To prevent compatibility issues, only use components that match the specifications and/or part numbers given.

Installation or replacement of most components require that power first be removed from the system. Please follow the procedures given in each section.

3.1 Removing Power

Before performing some setup or maintenance tasks, use the following procedure to ensure that power has been removed from the system.

Removing Power from a Node

- Use the operating system to power down the node.

Removing Power from the System

1. Use the operating system to power down all nodes.
2. Grasp the head of each power cord and gently pull it out of the back of the power supply.
3. Disconnect the cords from the power strip or wall outlet.

3.2 Accessing the System

The chassis contains eight computing nodes, each removable from the rear of the chassis while the others continue operating.

Removing a Computing Node

The two storage drives controlled by the node will power down upon removal.

1. Power down the node using the operating system or the power button.
2. Remove any cables attached to the node.
3. Push down and hold the release latch, then pull he node handle.
4. Slide the node out the chassis rear.

Release Latch Node Handle

Figure 3-1. Removing a Node

Removing the Chassis Cover

You can access some chassis components, such as fans, by removing the cover.

Screws

Figure 3-2. Removing the Chassis Cover

Removing the Chassis Cover

The chassis top cover can be lifted off after removing three screws.

Caution: Except for short periods of time, do not operate the server without the cover in place. It provides proper airflow to prevent overheating.

3.3 Motherboard Components

The processors (CPUs) and heatsinks are installed by the manufacturer.

Memory

Memory Support

The X11SDD-8C-F motherboard supports up to 256GB Registered ECC RDIMM and up to 512GB LRDIMM in four slots. Check the Supermicro website for possible updates to memory support.

DIMM Module Population Configuration

For optimal memory performance, follow the table below when populating memory.

Memory Population (Balanced)
DIMMA1	DIMMB1	DIM	MD1 DIMME1	Total System Memory
8GB 8GB	16GB
8GB 8GB	8GB 8GB	32GB
16GB 16GB	32GB
16GB 16GB	16GB 16GB	64GB
32GB 32GB	64GB
32GB 32GB	32GB 32GB	128GB
64GB 64GB	128GB
64GB 64GB	64GB 64GB	256GB

Memory Population Guidelines

• All DIMMs must be DDR4.
- Balance memory. Using unbalanced memory topology, such as populating two DIMMs in one channel while populating one DIMM in another channel, reduces performance. It is not recommended for Supermicro systems.
- Populating slots with a pair of DIMM modules of the same type and size results in interleaved memory, which improves memory performance.
- Use memory modules of the same type and speed, as mixing is not allowed.
- x4 and x8 DIMMs can be mixed in the same channel.
- Mixing of LRDIMMs and RDIMMs is not allowed in the same channel, across different channels, and across different sockets.
- Mixing of non-3DS and 3DS LRDIMM is not allowed in the same channel, across different channels, and across different sockets.

Guidelines Regarding Mixing DIMMs

DIMM Construction

• RDIMM (non-3DS) Raw Cards: A/B (2Rx4), C (1Rx4), D (1Rx8), E (2Rx8)
  • 3DS RDIMM Raw Cards: A/B (4Rx4)
  • LRDIMM (non-3DS) Raw Cards: D/E (4Rx4)
  • 3DS LRDIMM Raw Cards: A/B (8Rx4)

Memory Population Sequence

When installing memory modules, the DIMM slots should be populated in the following order: DIMMA1, DIMMB1, DIMMD1, DIMME1.

Blue slots versus black slots: Install the first DIMM in the blue memory slot, which is the first of a memory channel. Then, if using two DIMMs per channel, install the second DIMM in the black slot.

LED2 J3 M.2 H J2 M.2 H JRK1 SRW10 SRW6 BMC_HB_LED1 5RW11 SRW7 JWD7 5RW12 SRW8 IPME2 5RW5 SRW9 JPT1 BT1 CPU CPU SLOT PHE 3.0 X6 IPMI CODE MH7 MH9 JBT1 DIMMD1 DIMME1 DIMMA1 DIMMB1 SUPER X11SDD-8C-F REV.1.02 DESIGNED IN USA FCE CE DHW10 HDW2 DHW3 FAN1 MH8 S&PLO10 SASO MH1 JPWR1

Figure 3-3. DIMM Slots

Installing Memory

ESD Precautions

Electrostatic Discharge (ESD) can damage electronic components including memory modules. To avoid damaging DIMM modules, it is important to handle them carefully. The following measures are generally sufficient.

• Use a grounded wrist strap designed to prevent static discharge.
• Handle the memory module by its edges only.
• Put the memory modules into the antistatic bags when not in use.

Installing Memory

Begin by removing power from the system as described in Section 3.1. Follow the memory population sequence in the table above.

1. Push the release tabs outwards on both ends of the DIMM slot to unlock it.

Notches Release Tabs

1. Align the key of the DIMM with the receptive point on the memory slot and with your thumbs on both ends of the module, press it straight down into the slot until the module snaps into place.

Key

1. Press the release tabs to the locked position to secure the DIMM module into the slot.

Caution: Exercise extreme caution when installing or removing memory modules to prevent damage to the DIMMs or slots.

Removing Memory

To remove a DIMM, unlock the release tabs then pull the DIMM from the memory slot.

Motherboard Battery

The motherboard uses non-volatile memory to retain system information when system power is removed. This memory is powered by a lithium battery residing on the motherboard.

Replacing the Battery

Begin by removing power from the system as described in section 3.1.

1. Push aside the small clamp that covers the edge of the battery. When the battery is released, lift it out of the holder.
2. To insert a new battery, slide one edge under the lip of the holder with the positive (+) side facing up. Then push the other side down until the clamp snaps over it.

Note: Handle used batteries carefully. Do not damage the battery in any way; a damaged battery may release hazardous materials into the environment. Do not discard a used battery in the garbage or a public landfill. Please comply with the regulations set up by your local hazardous waste management agency to dispose of your used battery properly.

LITHIUM BATTERY BATTERY HOLDER

Figure 3-4. Installing the Onboard Battery

Warning: There is a danger of explosion if the onboard battery is installed upside down (which reverses its polarities). This battery must be replaced only with the same or an equivalent type recommended by the manufacturer (CR2032).

3.4 Chassis Components

This section provides instructions on installing and replacing system components. To assure compatibility, only use components that match the specifications or part numbers given.

Corresponding Nodes, Fans and Hard Drives

The SC938BH-R1620B chassis contains eight individual nodes each containing a motherboard. Each node controls two storage drives and shares a fan with the node beside it. If a node is pulled out of the chassis, the associated drives will power-down.

Fan 1 Fan 2 Fan 3 Fan 4 Node 1 Node 2 Node 3 Node 4 Node 5 Node 6 Node 7 Node 8 A B C D E F G H

Figure 3-5. Corresponding Nodes, Fans and Drives

Node Fan Drives
Node 1 Fan 1 Storage drives	A1 and A2
Node 2 Fan 1 Storage drives	B1 and B2
Node 3 Fan 2 Storage drives	C1 and C2
Node 4 Fan 2 Storage drives	D1 and D2
Node 5 Fan 3 Storage drives	E1 and E2
Node 6 Fan 3 Storage drives	F1 and F2
Node 7 Fan 4 Storage drives	G1 and G2
Node 8 Fan 4 Storage drives	H1 and H2

Storage Drives

The system supports sixteen 3.5" hot-swap storage drives, SATA3 by default. Optional adapter kits allow 2.5" NVMe/SATA drives instead. The drives are mounted in drive carriers that simplify their removal from the chassis. These carriers also help promote proper airflow. Even carriers without drives must remain in the chassis for proper airflow.

Drive Carrier Indicators

Each drive carrier has two LED indicators: an activity indicator and a status indicator. For RAID configurations using a controller, the meaning of the status indicator is described in the table below. For OS RAID or non-RAID configurations, some LED indications are not supported, such as hot spare. For VROC configurations, refer to the VROC appendix in this manual.

Drive Carrier LED Indicators
	Color Blinking Pattern Behavior for Device
Activity LED	Blue Solid On SAS/NVMe drive installed
	Blue Blinking I/O activity
Status LED	Red Solid On Failure of drive with RSTe support
	Red Blinking at 1 Hz Rebuild drive with RSTe support
	Red Blinking with two blinks and one stop at 1 Hz		Hot spare for drive with RSTe support (not supported in VMD mode)
	Red On for five seconds, then off		Power on for drive with RSTe support
	Red Blinking at 4 Hz Identify drive with RSTe support
	Green Solid On Safe to remove NVMe device (not supported in VMD mode)
	Amber Blinking at 1 Hz Attention state—do not remove NVMe device (not supported in VMD mode)

Note: Enterprise level hard disk drives are recommended for use in Supermicro chassis and servers. For information on recommended HDDs, visit the Supermicro website at https://www.supermicro.com/products/nfo/files/storage/SBB-HDDCompList.pdf.

Removing a Hot-Swap Drive Carrier from the Chassis

1. Press the release button on the drive carrier, which will extend the drive carrier handle.
2. Use the drive carrier handle to pull the drive out of the chassis.

Technical diagram showing server rack and tower structure with labeled components and directional arrows indicating assembly or rotation.

Figure 3-6. Removing a Drive Carrier

Installing a Drive

1. Remove the dummy drive, which comes pre-installed in the drive carrier, by removing the screws securing the dummy drive to the carrier. These screws are not used to mount the actual drive.

Dummy Drive Drive Carrier

Figure 3-7. Removing the Dummy Drive from a Carrier

1. Insert a drive into the carrier with the PCB side facing down and the connector end toward the rear of the carrier. Align the drive in the carrier so that the screw holes line up. Note that there are holes in the carrier marked "SATA" to aid in correct installation.
2. Secure the drive to the carrier with four M3 screws as illustrated below. These screws are included in the chassis accessory box.
3. Insert the drive carrier with the disk drive into its bay, keeping the carrier oriented so that the hard drive is on the top of the carrier and the release button is on the right side. When the carrier reaches the rear of the bay, the release handle will retract.
4. Push the handle in until it clicks into its locked position

Hot-Swap for NVMe Drives

Supermicro servers support NVMe surprise hot-swap. For even better data security, NVMe orderly hot-swap is recommended. NVMe drives can be ejected and replaced remotely using IPMI.

Note: If you are using VROC, see the VROC appendix in this manual instead.

Ejecting a Drive

1. IPMI > Server Health > NVMe SSD
2. Select Device, Group and Slot, and click Eject. After ejecting, the drive Status LED indicator turns green.
3. Remove the drive.

Note that Device and Group are categorized by the CPLD design architecture.

Slot is the slot number on which the NVMe drives are mounted.

Host Identification Server: 172.031.048.114 User: ADMIN (Administrator) System Server Health Configuration Remote Control Virtual Media Maintenance Miscellaneous Help Server Health Sensor Readings Health Event Log Power Consumption Power Source NVMe SSD NVMe SSD This page displays NVMe SSD information. Locale / Stop Locate SSD Devices:0 Groucp0 Slot:9 Locate Stop Locate End CPLD / BPN ID: EF Rev: 10 Slot:0 Status Present Temperature 35 degrees C Vendor ID 80.86 Serial Number BTLF7270687N4P0IGN Model Number INTEL SSDPE2X040T7 Slot:1 Status Present Temperature 33 degrees C Vendor ID 80.86 Serial Number PHLF729690684P0IGN Model Number INTEL SSDPE2X040T7 Slot:2 Copyright © 2016 Topo Micro Computer, Inc.

Figure 3-8. IPMI Screenshot

Replacing the Drive

1. Insert the replacement drive.
2. IPMI > Server Health > NVMe SSD
3. Select Device, Group and slot and click Insert. The drive Status LED indicator flashes red, then turns off. The Activity LED turns blue.

Checking the Temperature of an NVMe Drive

There are two ways to check using IPMI.

Checking a Drive

• IPMI > Server Health > NVMe SSD – Shows the temperatures of all NVMe drives, as in Figure 3-8.
• IPMI > Server Health > Sensor Reading > NVME_SSD – Shows the single highest temperature among all the NVMe drives.

M.2 Solid State Drives

Up to two M.2 solid state drives (SSDs) can be installed in socket J2 and J3 (see Section 1.5)

• M.2 Interface: 2 PCI-E 3.0 x4
• M.2 Form Factor: 2280/22110
• M.2 Key: M-Key

Installing an M.2 SSD

Caution: Use industry-standard anti-static equipment, such as gloves or wrist strap, and follow precautions to avoid damage caused by ESD.

Locate the socket J2 and J3 on the motherboard. There is a plastic standoff in one of the SRW holes. If it is the correct hole for your M.2 SSD, you can slide the SSD into the socket, and secure it by pushing the plug into the standoff. If not, you must move the plastic standoff.

Standoff Plug

Figure 3-9. Inserting the Standoff Plug
(Note: Your card looks different, but the standoff functions the same.)

System Cooling

Fans

The chassis contains four 8-cm high-performance fans. Fan speed is controlled by IPMI depending on the system temperature. If a fan fails, the remaining fans will ramp up to full speed. The system will continue to run with a failed fan, although it may shut down if the heat gets too great. Replace any failed fan at your earliest convenience with the same model. Failed fans can be identified through the IPMI.

Changing a System Fan

1. Determine which fan has failed using IPMI, or if necessary, open the chassis while the system is running. Never run the server for long without the chassis cover.

2. Squeeze both release tabs on the top of the fan module and pull it out.

3. Replace the failed fan with an identical fan, available from Supermicro. Push the new fan into the housing, making sure the air flow direction is the same.

4. Power up the system and check that the fan is working properly and that the LED on the control panel has turned off. Finish by replacing the chassis cover.

Technical line drawing of a server rack with multiple drive bays and a central tower (no text or labels)

Figure 3-10. Replacing a Fan

Installing the Air Shroud

Air shrouds and air blocks concentrate airflow to maximize fan efficiency. Air block is pre-installed and may be uninstalled by removing the two screws.

Installing the Air Shroud

1. Power down the node and remove it. If an expansion card is used, install it first.

2. Place the Mylar air shroud over the two DIMMs, as shown below. The air shroud sits behind the CPU heatsink and goes over the top of the DIMMs.

Mylar Air Shroud DIMMs Air Block and Screws Node

Figure 3-11. Installing the Air Shroud

Power Supply

The system features redundant power supplies. The system will continue to operate if one module fails. It should be replaced as soon as convenient. They can be changed without powering down the system. New units can be ordered directly from Supermicro or authorized distributors.

These power supplies are auto-switching capable. This feature enables them to automatically sense the input voltage and operate at a 100-120v or 180-240v.

Power Supply LEDs

On the rear of the power supply module, an LED displays the status.

• Solid Green: When illuminated, indicates that the power supply is on.
• Blinking Green: When blinking, indicates that the power supply is plugged in and turned off by the system.
• Blinking Amber: When blinking, indicates that the power supply has a warning condition and continues to operate.
• Solid Amber: When illuminated, indicates that the power supply is plugged in, and is in an abnormal state. The system might need service. Please contact Supermicro technical support.

Release Tab

Figure 3-12. Replacing the Power Supply

Changing the Power Supply Module:

1. Unplug the AC cord from the module to be replaced.
2. On the back of the module, push the release tab, and pull the module out using the handle.
3. Push the new power supply module into the power bay until it clicks. Replace with the same model.
4. Plug the AC power cord back into the module.

PCI Expansion Cards

The system supports one low-profile PCI-E card in each of the eight nodes with the included riser card (RSC-RR1U-E16).

Installing an Expansion Card

1. Power-down the node and remove it.
2. Remove the two screws that secure the PCI bracket to the motherboard tray.
3. Remove the PCI slot bracket.
4. Remove the PCI slot shield.

PCI Slot Shield

Figure 3-13. PCI Expansion Card Chassis Slots

1. Insert the expansion card into the riser card, which is pre-installed on the motherboard (if desired the riser card may be removed from the motherboard by removing the riser card screw).
2. Slide the add-on cards bracket into the PCI card slot and fit it with the opening in the rear of the node.
3. Close the PCI card slot clip to secure the add-on card.

Screws PCI Slot Bracket Riser Card GPU Air Shroud Expansion Card Motherboard Tray Node

Figure 3-14. Installing an Expansion Card Chassis Slots

Chapter 4

Motherboard Connections

This section describes the connections on the motherboard and provides pinout definitions.

Note that depending on how the system is configured, not all connections are required.

The LEDs on the motherboard are also described here. A motherboard layout indicating component locations may be found in Chapter 1.

Please review the safety precautions in Appendix B before installing or removing components.

4.1 Power Connections

Power Connector

Connect a 2-pin power supply cable to JPWR1 to provide power to the motherboard.

Power ConnectorPin Definitions
Pin#	Definition
1	Ground
2	P12V

4.2 Headers and Connectors

Fan Headers

This motherboard has one fan header (FAN1). IT is a 4-pin fan headers, although pins 1-3 are backward compatible with traditional 3-pin fans. Four-pin fans allow fan speeds to be controlled by Thermal Management in IPMI. When using the Thermal Management setting, use all 3-pin fans or all 4-pin fans.

Fan HeaderPin Definitions
Pin#	Definition
1	Ground (Black)
2	+12V (Red)
3	Tachometer
4	PWM Control

TPM Header

The JTPM1 header is used to connect a Trusted Platform Module (TPM)/Port 80, which is available from Supermicro. A TPM/Port 80 connector is a security device that supports encryption and authentication in hard drives. It allows the motherboard to deny access if the TPM associated with the hard drive is not installed in the system. For more information, http://www.supermicro.com/manuals/other/TPM.pdf.

Trusted Platform Module/Port 80 Header Pin Definitions
Pin# Definition Pin# Definition
1 P3V3 2 SPI_TPM_CS_N
3 PCI-E_RESET_N# 4 SPI_PCH_MISO
5 SPI_PCH_CLK# 6 Ground
7 SPI_PCH_MOSI 8 N/A
9 JTPM1_P3V3A 10 IRQ_TPM_SPIN_N

SGPIO Header

A Serial General Purpose Input/Output header (T-SGPIO1) communicates with the enclosure management chip on the backplane.

S-SGPIO HeaderPin Definitions
Pin# Definition Pin# Definition
1 NC 2 NC
3 Ground 4 Data
5 Load 6 Ground
7 Clock 8 NC

NC = No Connection

RAID Key Header

A RAID Key header is located at JRK1 on the motherboard. It supports VMD used in creating optional advanced NVMe RAID configurations.

Intel RAID KeyPin Definitions
Pin#	Definition
1	Ground
2	3.3V Standby
3	Ground
4	PCH RAID Key

4.3 Input/Output Ports

Rear I/O Ports

The rear of each node offers two Gb Ethernet ports, control switches and a KVM port.

Close-up of electronic device ports including a D-sub connector, a circular connector, and a small electronic component (no visible text or symbols)

Figure 4-2. Rear I/O Ports

# Description

1 KVM Connector

2 Power Switch

3 UID Switch

Three black and blue cables with connectors, no visible text or symbols

Figure 1-5. Adapter to Provide USB, COM, and VGA Ports

KVM port

With a KVM dongle, each node offers two USB 2.0 ports, one VGA port and one COM port.

Unit Identifier Switch/UID LED Indicator

A Unit Identifier (UID) switch and an LED Indicator are located on the back panel. When you press the UID switch, both front and rear UID LED indicators will be turned on. Press the UID switch again to turn off the LEDs. The UID indicators provide easy identification of a system in a rack.

Note: UID can also be triggered using IPMI.

Ethernet Ports

Two Ethernet ports are provided by means of an add-on module mounted on the Micro-LP expansion slot.

4.4 Jumpers

Explanation of Jumpers

To modify the operation of the motherboard, jumpers are used to choose between optional settings. Jumpers create shorts between two pins to change the function associated with it. Pin 1 is identified with a square solder pad on the printed circuit board. See the motherboard layout page for jumper locations.

Note: On a two-pin jumper, "Closed" means the jumper is on both pins and "Open" indicates the jumper is either on only one pin or has been completely removed.

Connector Pins Jumper Setting 3 2 1 ● ● ■ 3 2 1

CMOS Clear

JBT1 is used to clear CMOS, which will also clear any passwords. Instead of pins, this jumper consists of contact pads to prevent accidentally clearing the contents of CMOS.

To Clear CMOS

1. First power down the system and unplug the power cord(s).
2. Remove the cover of the chassis to access the motherboard.
3. Remove the onboard battery from the motherboard.
4. Short the CMOS pads with a metal object such as a small screwdriver for at least four seconds.
5. Remove the screwdriver (or shorting device).
6. Replace the cover, reconnect the power cord(s) and power on the system.

Notes: Clearing CMOS will also clear all passwords.

Do not use the PW_ON connector to clear CMOS.

JBT1 contact pads

Watch Dog

JWD1 controls the Watch Dog function. Watch Dog is a monitor that can reboot the system when a software application hangs. Jumping pins 1-2 will cause Watch Dog to reset the system if an application hangs. Jumping pins 2-3 will generate a non-maskable interrupt signal for the application that hangs. Watch Dog must also be enabled in BIOS. The default setting is Reset.

Note: When Watch Dog is enabled, the user must write their own application software to disable it.

Watch DogJumper Settings
Jumper Setting Definition
Pins 1-2 Reset
Pins 2-3 NMI
Open Disabled

Manufacturing Mode Select

Close JPME2 to bypass SPI flash security and force the system to use the Manufacturing Mode, which will allow you to flash the system firmware from a host server to modify system settings.

Manufacturing Mode SelectJumper Settings
Jumper Setting Definition
Pins 1-2 Normal	(Default)
Pins 2-3 Manufacturing Mode

4.5 LED Indicators

LAN LEDs

The Ethernet ports each have two LEDs. One LED indicates activity when flashing, while the other LED may be green, amber or off to indicate the speed of the connection.

LAN LED(Connection Speed Indicator)
LED Color Definition
Off 10 Mb/s
Green 25 Gb/s
Amber 1 Gb/s

Dedicated IPMI LAN LEDs

A dedicated IPMI LAN port is on the I/O back panel. The amber LED on the right of the IPMI LAN port indicates activity, while the LED on the left indicates the link speed of the connection.

IPMI LAN Activity LED (Right)
LED Color/State Definition
Activity (Right) Green: Blinking 100 Mbps
Activity (Right) Amber: Blinking 1G

Unit ID LED

A rear unit identifier (UID) indicator at LE2 is located near the UID switch on the I/O back panel. It provides easy identification of a unit in a rack.

System Failure LED

LED3 indicates an overheated system, a power supply failure or fan failure.

System Overheat/Power Failure/FanFailure LED Indicator

LED Color Definition

Solid Red System Overheat

Blinking Red Power or Fan Failure

Off Normal operation

BMC Heartbeat LED

BMC_HB_LED1 is the BMC heartbeat LED. When the LED is blinking green, BMC is functioning normally.

Chapter 5

Software

After the hardware has been installed, you can install the Operating System (OS), configure RAID settings and install the drivers.

5.1 Microsoft Windows OS Installation

If you will be using RAID, you must configure RAID settings before installing the Windows OS and the RAID driver. Refer to the RAID Configuration User Guides posted on our website at www.supermicro.com/support/manuals.

Installing the OS

1. Create a method to access the MS Windows installation ISO file. That might be a DVD, perhaps using an external USB/SATA DVD drive, or a USB flash drive, or the IPMI KVM console.
2. Retrieve the proper RST/RSTe driver. Go to the Supermicro web page for your motherboard and click on "Download the Latest Drivers and Utilities", select the proper driver, and copy it to a USB flash drive.
3. Boot from a bootable device with Windows OS installation. You can see a bootable device list by clicking F11 during the system startup.

Please select boot device: ATEN Virtual CDROM YSOJ → IPMI virtual drive (Legacy) ASUS SDRW-08D2S-U F601 → USB DVD device (Legacy) USB FLASH DRIVE PMAP → USB flash drive with OS installation (Legacy) IBA 40-10G Slot 1900 v1060 → PXE boot (Legacy) UEFI: ATEN Virtual CDROM YSOJ → IPMI virtual drive (UEFI) UEFI: ASUS SDRW-08D2S-U F601 → USB DVD device (UEFI) UEFI: Built-in EFI Shell Enter Setup ↑ and ↓ to move selection ENTER to select boot device ESC to boot using defaults

Figure 5-1. Select Boot Device

1. During Windows Setup, continue to the dialog where you select the drives on which to install Windows. If the disk you want to use is not listed, click on "Load driver" link at the bottom left corner.

Where do you want to install Windows? Name Total size Free space Type Refresh Delete Format New Load driver Extend We couldn't find any drives. To get a storage driver, click Load driver. Next

Figure 5-2. Load Driver Link

To load the driver, browse the USB flash drive for the proper driver files.

• For RAID, choose the SATA/sSATA RAID driver indicated then choose the storage drive on which you want to install it.

• For non-RAID, choose the SATA/sSATA AHCI driver indicated then choose the storage drive on which you want to install it.

• Once all devices are specified, continue with the installation.

• After the Windows OS installation has completed, the system will automatically reboot multiple times.

5.2 Driver Installation

The Supermicro website contains drivers and utilities for your system at https://www.supermicro.com/wftp/driver. Some of these must be installed, such as the chipset driver.

After accessing the website, go into the CDR_Images (in the parent directory of the above link) and locate the ISO file for your motherboard. Download this file to a USB flash drive or a DVD. (You may also use a utility to extract the ISO file if preferred.)

Another option is to go to the Supermicro website at http://www.supermicro.com/products/. Find the product page for your motherboard, and "Download the Latest Drivers and Utilities". Insert the flash drive or disk and the screenshot shown below should appear.

SUPERMICRO X11SDD-8C-F Motherboard Drivers & Tools (Win10) Intel Skylake-D Chipset X11SDD-8C/ 18C-F MB SUPERMICRO Computer Inc. Intel Chipset INF files Microsoft .Net Framework 4.5.2(Optional) ASPEED Graphics Driver Intel Rapid Storage Technology Enterprise Intel USB 3.0 Drivers Intel PRO Network Connections Drivers SUPERMICRO SuperDoctor 5 Build driver diskettes and manuals Browse CD Auto Start Up Next Time For more information, please visit SUPERMICRO's web site.

Figure 5-3. Driver & Tool Installation Screen

Note: Click the icons showing a hand writing on paper to view the readme files for each item. Click the computer icons to the right of these items to install each item (from top to the bottom) one at a time. After installing each item, you must re-boot the system before moving on to the next item on the list. The bottom icon with a CD on it allows you to view the entire contents.

5.3 SuperDoctor® 5

The Supermicro SuperDoctor 5 is a program that functions in a command-line or web-based interface for Windows and Linux operating systems. The program monitors such system health information as CPU temperature, system voltages, system power consumption, fan speed, and provides alerts via email or Simple Network Management Protocol (SNMP).

SuperDoctor 5 comes in local and remote management versions and can be used with Nagios to maximize your system monitoring needs. With SuperDoctor 5 Management Server (SSM Server), you can remotely control power on/off and reset chassis intrusion for multiple systems with SuperDoctor 5 or IPMI. SuperDoctor 5 Management Server monitors HTTP, FTP, and SMTP services to optimize the efficiency of your operation.

Note: The default User Name and Password for SuperDoctor 5 is ADMIN / ADMIN.

SuperDoctor 5 Home info: www.humanlab.com Certificate error: localhost:1044/superdoctor SuperDoctor 5 Select Language: English (ADMIN) Language Motherboard: C78366-CB-ML Voltage 0.0V 12.0V 14.0V 16.0V 18.0V 20.0V 22.0V 24.0V 26.0V 28.0V 30.0V 32.0V 34.0V 36.0V 38.0V 40.0V 42.0V 44.0V 46.0V 48.0V 50.0V 52.0V 54.0V 56.0V 58.0V 60.0V 62.0V 64.0V 66.0V 68.0V 70.0V 72.0V 74.0V 76.0V 78.0V 80.0V 82.0V 84.0V 86.0V 88.0V 90.0V 92.0V 94.0V 96.0V 98.0V 100.0V 102.0V 104.0V 106.0V 108.0V 110.0V 112.0V 114.0V 116.0V 118.0V 120.0V 122.0V 124.0V 126.0V 128.0V 130.0V 132.0V 134.0V 136.0V 138.0V 140.0V 142.0V 144.0V 146.0V 148.0V 150.0V 152.0V 154.0V 156.0V 158.0V 160.0V 162.0V 164.0V 166.0V 168.0V 170.0V 172.0V 174.0V 176.0V 178.0V 180.0V 182.0V 184.0V 186.0V 188.0V 190.0V 192.0V 194.0V 196.0V 198.0V 200.0V 202.0V 204.0V 206.0V 208.0V 210.0V 212.0V 214.0V 216.0V 218.0V 220.0V 222.0V 224.0V 226.0V 228.0V 230.0V 232.0V 234.0V 236.0V 238.0V 240.0V

Figure 5-4. SuperDoctor 5 Interface Display Screen (Health Information)

5.4 IPMI

The X11SDD-8C-F supports the Intelligent Platform Management Interface (IPMI). IPMI is used to provide remote access, monitoring and management. There are several BIOS settings that are related to IPMI.

IPMI management capabilities for all nodes are supported when one of two dedicated IPMI LAN ports are connected (Figure 1-3). Multi-node information and the system information can be displayed at the same time.

In IPMI Web GUI view, information can be displayed by clicking Multi Node under the Server Health drop-down menu. To access the other nodes within the same enclosure, click the IP address of the other nodes.

In IPMI CLI view, typing the command "SMCIPMITool -tp info", allows the MCU information of all eight nodes to be displayed.

For general documentation and information on IPMI, please visit our website at: http://www.supermicro.com/products/nfo/IPMI.cfm.

Chapter 6

BIOS

6.1 Introduction

This chapter describes the AMI BIOS setup utility for the X11SDD-8C-F and provides the instructions on navigating the setup screens. The BIOS is stored in a Flash EEPROM and can be updated.

Note: Due to periodic changes to the BIOS, some settings may have been added or deleted since this manual was published.

Starting BIOS Setup Utility

To enter the AMI BIOS setup utility screens, press the key while the system is booting up. (There are a few cases when other keys are used, such as , , etc.)

The BIOS screens have three main frames. The large left frame displays options can be configured by the user. These are blue. When an option is selected, it is highlighted in white. Settings printed in Bold are the default values.

In the left frame, a "▶" indicates a submenu. Highlighting such an item and pressing the key opens the list of settings in that submenu.

The upper right frame displays helpful information for the user. The AMI BIOS has default informational messages built in. The manufacturer retains the option to include, omit, or change any of these informational messages.

The lower right frame lists navigational methods. The AMI BIOS setup utility uses a key-based navigation system called hot keys. Most of these hot keys can be used at any time during setup navigation. These keys include , , , , arrow keys, etc.

Some system parameters may be changed.

6.2 Main Setup

When running the AMI BIOS setup utility, it starts with the Main screen. You can always return to it by selecting the Main tab on the top of the screen.

Aptio Setup Utility - Copyright (C) 2018 American Megatrends, Inc. Main Advanced IPMI Security Event Logs Boot Save & Exit System Date [Sat 12/08/2018] System Time [23:21:37] Supermicro X11SDD BIOS Version 1.0 Build Date 07/24/2018 CPLD Version 04.b0.01 Memory Information Total Memory 8192 MB Memory Speed 2132 MT/s Set the Date. Use Tab to switch between Date elements. +: Select Screen ↑↓: Select Item Enter: Select +/-: Change Opt. F1: General Help F2: Previous Values F3: Optimized Defaults F4: Save & Exit ESC: Exit Version 2.20.1271. Copyright (C) 2018 American Megatrends, Inc.

The Main tab page allows you to set the date and time, and it displays system information.

System Date/System Time

Use this option to change the system date and time. Highlight System Date or System Time using the arrow keys. Enter new values using the keyboard. Press the key or the arrow keys to move between fields. The date must be entered in MM/DD/YYYY format. The time is entered in HH:MM:SS format.

Note: The time is in the 24-hour format. For example, 5:30 P.M. appears as 17:30:00. The date's default value is 01/01/2016 after RTC reset.

Supermicro X11SDD-8C-F (Motherboard model)

BIOS Version

Build Date (of the BIOS)

CPLD (Complex Programmable Logic Device) Version: This item displays the CPLD version used in the system.

Memory Information

Total Memory (for the system)

Memory Speed

6.3 Advanced Setup Configurations

Use the arrow keys to select the Advanced tab and press to access the submenu items.

Aptio Setup Utility - Copyright (C) 2018 American Megatrends, Inc. Main Advanced IPMI Security Event Logs Boot Save & Exit
Boot Feature CPU Configuration Chipset Configuration Server ME Information SATA Configuration sSATA Configuration PCIe/PCI/PnP Configuration Super IO Configuration Serial Port Console Redirection ACPI Settings Trusted Computing Tls Auth Configuration Intel(R) Virtual RAID on CPU All Cpu Information	Display all cpu information
	+: Select Screen ↑↓: Select Item Enter: Select +/-: Change Opt. F1: General Help F2: Previous Values F3: Optimized Defaults F4: Save & Exit ESC: Exit
Version 2.20.1271. Copyright (C) 2018 American Megatrends, Inc.

Caution: Take caution when changing the Advanced settings. An incorrect value, a very high DRAM frequency, or an incorrect DRAM timing setting may make the system unstable. If this occurs, revert to the manufacture default settings.

▶Boot Feature

Quiet Boot

Use this feature to select the screen display between POST messages or the OEM logo at bootup. Select Disabled to display the POST messages. Select Enabled to display the OEM logo instead of the normal POST messages. The options are Disabled and Enabled.

Option ROM Messages

Use this feature to set the display mode for the Option ROM. Select Keep Current to display the current AddOn ROM setting. Select Force BIOS to use the Option ROM display set by the system BIOS. The options are Force BIOS and Keep Current.

Bootup NumLock State

Use this feature to set the Power-on state for the Numlock key. The options are On and Off.

Wait For "F1" If Error

This feature forces the system to wait until the F1 key is pressed if an error occurs. The options are Disabled and Enabled.

Interrupt 19 Capture

Interrupt 19 is the software interrupt that handles the boot disk function. When this feature is set to Immediate, the ROM BIOS of the host adaptors will "capture" Interrupt 19 at boot up immediately and allow the drives that are attached to these host adaptors to function as bootable disks. If this feature is set to Postponed, the ROM BIOS of the host adaptors will not capture Interrupt 19 immediately and allow the drives attached to these adapters to function as bootable devices at bootup. The options are Immediate and Postponed.

Re-try Boot

If this feature is enabled, the BIOS will automatically reboot the system from a specified boot device after its initial boot failure. The options are Disabled, Legacy Boot, and EFI Boot.

Power Configuration

Watch Dog Function

If enabled, the Watch Dog timer will allow the system to reboot when it is inactive for more than five minutes. The options are Disabled and Enabled.

AC Loss Policy Depend on

Use this feature to set the power state after a power outage. Select Power Off for the system power to remain off after a power loss. Select Power On for the system power to be turned on after a power loss. Select Last State to allow the system to resume its last power state before a power loss. The options are Stay Off, Power On, and Last State.

Power Button Function

This feature controls how the system shuts down when the power button is pressed. Select 4 Seconds Override for the user to power off the system after pressing and holding the power button for four seconds or longer. Select Instant Off to instantly power off the system as soon as the user presses the power button. The options are Instant Off and 4 Seconds Override.

Restore On AC Power Loss

Use this feature to set the power state after a power outage. Select Stay Off for the system power to remain off after a power loss. Select Power On for the system power to be turned on after a power loss. Select Last State to allow the system to resume its last power state before the power loss. The options are Power On, Stay Off, and Last State.

Throttle on Power Fail

Use this feature to decrease system power by throttling CPU frequency when one power supply has failed. The options are Disabled and Enabled.

▶CPU Configuration

The following CPU information will display:

• Processor BSP Revision
• Processor Socket
• Processor ID
• Processor Frequency
• Processor Max Ratio
• Processor Min Ratio
• Microcode Revision
• L1 Cache RAM
• L2 Cache RAM
• L3 Cache RAM
• Processor 0 Version

Hyper-Threading [ALL] (Available when supported by the CPU)

Select Enable to support Intel Hyper-threading Technology to enhance CPU performance. The options are Disable and Enable.

Cores Enabled

Set a numeric value to enable the number of cores. Enter 0 to enable all cores.

Execute Disable Bit (Available if supported by the OS & the CPU)

Set to Enable for Execute Disable Bit support, which will allow the processor to designate areas in the system memory where an application code can execute and where it cannot, thus preventing a worm or a virus from flooding illegal codes to overwhelm the processor or damaging the system during a virus attack. The options are Disable and Enable. Refer to Intel and Microsoft websites for more information.

Intel Virtualization Technology

Use this feature to enable the Vanderpool Technology. This technology allows the system to run several operating systems simultaneously. The options are Disabled and Enabled.

PPIN Control

Select Unlock/Enable to use the Protected-Processor Inventory Number (PPIN) in the system. The options are Unlock/Disable and Unlock/Enable.

Hardware Prefetcher

If set to Enable, the hardware prefetcher will prefetch streams of data and instructions from the main memory to the L2 cache to improve CPU performance. The options are Disabled and Enabled.

Adjacent Cache Line Prefetch

The CPU prefetches the cache line for 64 bytes if this feature is set to Disabled. The CPU prefetches both cache lines for 128 bytes as comprised if this feature is set to Enabled. The options are Disabled and Enabled.

DCU Streamer Prefetcher (Available when supported by the CPU)

Select Enable to enable the DCU (Data Cache Unit) Streamer Prefetcher, which will stream and prefetch data and send it to the Level 1 data cache to improve data processing and system performance. The options are Enable and Disable.

DCU IP Prefetcher (Available when supported by the CPU)

Select Enable for DCU (Data Cache Unit) IP Prefetcher support, which will prefetch IP addresses to improve network connectivity and system performance. The options are Enable and Disable.

LLC Prefetch

If set to Enable, the hardware prefetcher will prefetch streams of data and instructions from the main memory to the L3 cache to improve CPU performance. The options are Disable and Enable.

Extended APIC (Advanced Programmable Interrupt Controller)

Based on the Intel Hyper-Threading technology, each logical processor (thread) is assigned 256 APIC IDs (APIDs) in 8-bit bandwidth. When this feature is set to Enable, the APIC ID will be expanded from 8 bits to 16 bits to provide 512 APIDs to each thread to enhance CPU performance. The options are Disable and Enable.

AES-NI

Select Enable to use the Intel Advanced Encryption Standard (AES) New Instructions (NI) to ensure data security. The options are Disable and Enable.

▶Advanced Power Management Configuration

▶CPU P State Control

This feature allows the user to configure the following CPU power settings:

SpeedStep (Pstates)

Intel SpeedStep Technology allows the system to automatically adjust processor voltage and core frequency to reduce power consumption and heat dissipation. The options are Disable and Enable.

EIST PSD Function

This feature allows the user to choose between Hardware and Software to control the processor's frequency and performance (P-state). In HW_ALL mode, the processor hardware is responsible for coordinating the P-state, and the OS is responsible for keeping the P-state request up to date on all Logical Processors. In SW_ALL mode, the OS Power Manager is responsible for coordinating the P-state and must initiate the transition on all Logical Processors. In SW_ANY mode, the OS Power Manager is responsible for coordinating the P-state and may initiate the transition on any Logical Processors. The options are HW_ALL, SW_ALL, and SW_ANY.

Turbo Mode

This feature will enable dynamic control of the processor, allowing it to run above stock frequency. The options are Disable and Enable.

▶Hardware PM State Control

Hardware P-States

This setting allows the user to select between OS and hardware-controlled P-states. Selecting Native Mode allows the OS to choose a P-state. Selecting Out of Band Mode allows the hardware to autonomously choose a P-state without OS guidance. Selecting Native Mode with No Legacy Support functions as Native Mode with no support for older hardware. The options are Disable, Native Mode, Out of Band Mode, and Native Mode with No Legacy Support.

▶CPU C State Control

Autonomous Core C-State

Enabling this setting allows the hardware to autonomously choose to enter a C-state based on power consumption and clock speed. The options are Disable and Enable.

CPU C6 report

Select Enable to allow the BIOS to report the CPU C6 State (ACPI C3) to the operating system. During the CPU C6 State, the power to all cache is turned off. The options are Disable, Enable, and Auto.

Enhanced Halt State (C1E)

Select Enable to use Enhanced Halt State technology, which will significantly reduce the CPU's power consumption by reducing its clock cycle and voltage during a Halt state. The options are Disable and Enable.

▶Package C State Control

Package C State

This feature allows the user to set the limit on the C State package register. The options are C0/C1 State, C2 State, C6 (non Retention) state, C6 (Retention) state, No Limit, and Auto.

▶Chipset Configuration

Warning: Setting the wrong values in the sections below may cause the system to malfunction.

▶ North Bridge

This feature allows the user to configure the following North Bridge settings.

▶ Memory Configuration

Enforce POR

Select POR (Plan of Record) to enforce POR restrictions on DDR4 frequency and voltage programming. The options are POR and Disable.

Memory Frequency

Use this feature to set the maximum memory frequency for onboard memory modules. The options are Auto, 1000, 1200, 1333, 1400, 1600, 1800, 1866, 2000, 2133, 2200, 2400, 2600, and 2666.

IMC Interleaving

This feature allows the user to configure Integrated Memory Controller (IMC) Interleaving settings. The options are Auto, 1-way Interleave, and 2-way Interleave.

Data Scrambling for NVDIMM

Use this feature to enable or disable data scrambling support for NVDIMM. The options are Auto, Disable, and Enable.

Data Scrambling for DDR4

Use this feature to enable or disable data scrambling support for DDR4. The options are Auto, Disable, and Enable.

tCCD\_L Relaxation

Select Auto to get TCDD settings from SPD (Serial Presence Detect) and implement into memory RC code to improve system reliability. Select Disable for TCCD to follow Intel POR. The options are Auto, Disable, and Enable.

tRWSR Relaxation

Select Enable to override the tRWSR timing with optimized values. Select Auto to override the tWRSR timing for Samsung DIMM memory. The options are Auto, Disable, and Enable.

Enable ADR

Select Enable for ADR (Automatic Diagnostic Repository) support to enhance memory performance. The options are Disable and Enable.

2X REFRESH

Use this feature to select the memory controller refresh rate to 2x refresh mode. The options are Auto, Enable, and Disable.

Page Policy

This feature allows the user to determine the desired page mode for IMC. When Auto is selected, the memory controller will close or open pages based on the current operation. Closed policy closes that page after reading or writing. Adaptive is similar to open page policy, but can be dynamically modified. The options are Auto, Closed, and Adaptive.

▶Memory Topology

This item displays the information of onboard memory modules as detected by the BIOS.

▶ Memory RAS Configuration

Static Virtual Lockstep Mode

Select Enable to run the system's memory channels in lockstep mode to minimize memory access latency. The options are Disable and Enable.

Mirror Mode

This feature allows memory to be mirrored between two channels, providing 100% redundancy and consequently reducing the memory capacity by half. The options are Disable, Mirror Mode 1LM, and Mirror Mode 2LM.

UEFI ARM Mirror

Select Enable to support the UEFI-based address range mirroring with setup option. The options are Disable and Enable.

Memory Rank Sparing

Select Enable to enable memory-sparing support for memory ranks to improve memory performance. The options are Disable and Enable.

Correctable Error Threshold

Use this feature to specify the threshold value for correctable memory error logging, which sets a limit on the maximum number of events that can be logged in the memory error log at a given time. Select a value between 1-32776. The default setting is 7fff.

SDDC

Single device data correction (SDDC) organizes data in a single bundle (x4/x8 DRAM). If any or all the bits become corrupted, corrections occur. The x4 condition is corrected on all cases. The x8 condition is corrected only if the system is in Lockstep Mode. The options are Disable and Enable.

ADDDC Sparing

Adaptive Double Device Data Correction (ADDDC) Sparing detects when the predetermined threshold for correctable errors is reached, copying the contents of the failing DIMM to spare memory. The failing DIMM or memory rank will then be disabled. The options are Disable and Enable.

Patrol Scrub

Patrol Scrub is a process that allows the CPU to correct correctable memory errors detected on a memory module and send the correction to the requestor (the original source). When this item is set to Enable, the IO hub will read and write back one cache line every 16K cycles if there is no delay caused by internal processing. By using this method, roughly 64 GB of memory behind the IO hub will be scrubbed every day. The options are Disable and Enable.

Patrol Scrub Interval

This feature allows the user to decide how many hours the system should wait before the next complete patrol scrub is performed. Use the keyboard to enter a value from 0-24. The default setting is 24.

▶IIO Configuration

▶Socket0 Configuration

IOU0 (II0 PCIe Br1) / IOU1 (II0 PCIe Br2)

This features configure the PCI-E port Bifuraction setting for a PCI-E port specified by the user. The options are x4x4x4x4, x4x4x8, x8x4x4, x8x8, x16, and Auto.

MCP0 (II0 PCIe Br4) / MCP1 (II0 PCIe Br5)

This features configure the PCI-E port Bifuraction setting for a PCI-E port specified by the user. The options are x16 and Auto.

▶Socket 0 PcieBr0D00F0 - Port 0/DMI
▶Socket 0 PcieBr1D00F0 - Port 1A
▶Socket 0 PcieBr1D01F0 - Port 1B
▶Socket 0 PcieBr1D02F0 - Port 1C
▶Socket 0 PcieBr2D00F0 - Port A2
▶Socket 0 PcieBr4D00F0 - MCP 0
▶Socket 0 PcieBr5D00F0 - MCP 1

Link Speed

Use this feature to select the link speed for the PCI-E port specified by the user. The options are Auto, Gen 1 (2.5 GT/s), Gen 2 (5 GT/s), and Gen 3 (8 GT/s).

PCI-E Port Max Payload Size

Selecting Auto for this feature will enable the motherboard to automatically detect the maximum Transaction Layer Packet (TLP) size for the connected PCI-E device, allowing for maximum I/O efficiency. Selecting 128B or 256B will designate maximum packet size of 128 or 256. If possible, select max payload size to 256B. The options are 128B, 256B, and Auto.

ECRC

Use this feature to enable or disable Error Capabilities and Control Register (ECRC). The options are Disable and Enable.

▶IOAT Configuration

Disable TPH

Transparent Huge Pages (TPH) is a Linux memory management system that enables communication in larger blocks (pages). Enabling this feature will increase performance. The options are No and Yes.

Prioritize TPH

Use this feature to enable Prioritize TPH support. The options are Enable and Disable.

Relaxed Ordering

Select Enable to enable Relaxed Ordering support, which will allow certain transactions to violate the strict-ordering rules of PCI bus for a transaction to be completed prior to other transactions that have already been enqueued. The options are Disable and Enable.

▶Intel® VT for Directed I/O (VT-d)

Intel® VT for Directed I/O (VT-d)

Select Enable to use Intel Virtualization Technology for Direct I/O VT-d support by reporting the I/O device assignments to the VMM (Virtual Machine Monitor) through the DMAR ACPI tables. This feature offers fully protected I/O resource sharing across Intel platforms, providing greater reliability, security and availability in networking and data sharing. The options are Enable and Disable.

Interrupt Remapping

Use this feature to enable Interrupt Remapping support, which detects and controls external interrupt requests. The options are Enable and Disable.

PassThrough DMA

Use this feature to allow devices such as network cards to access the system memory without using a processor. Select Enable to use the Non-Isoch VT_D Engine Pass Through Direct Memory Access (DMA) support. The options are Enable and Disable.

ATS

Use this feature to enable Non-Isoch VT-d Engine Address Translation Services (ATS) support. ATS translates virtual addresses to physical addresses. The options are Enable and Disable.

Posted Interrupt

Use this feature to enable VT_D Posted Interrupt. The options are Enable and Disable.

Coherency Support (Non-Isoch)

Use this feature to maintain setting coherency between processors or other devices. Select Enable for the Non-Isoch VT-d engine to pass through DMA to enhance system performance. The options are Enable and Disable.

Intel® VMD Technology

Intel® VMD for Volume Management Device on Socket 0

VMD Config for PStack0

Intel® VMD for Volume Management Device for PStack0

Select Enable to use the Intel Volume Management Device Technology for this stack. The options are Disable and Enable.

*If the feature above is set to Enable, the following features will become available for configuration:

VMD port 1A/1B/1C/1D

Select Enable to use the Intel Volume Management Device Technology for this specific root port. The options are Disable and Enable.

Hot Plug Capable (Available when the device is detected by the system)

Use this feature to enable hot plug support for PCIe root ports 1A\~1D. The options are Disable and Enable.

CfgBar size

Use this feature to set up the VMD config BAR size. The default value is 25.

CfgBar attribute

Use this feature to set up the VMD config BAR attribute. The default option is 64-bit prefetchable.

MemBar1 size

Use this feature to set up the VMD Memory BAR1 size. The default value is 25.

MemBar1 attribute

Use this feature to set up the VMD config BAR attribute. The options are 32-bit non-prefetchable, 64-bit non-prefetchable, and 64-bit prefetchable.

MemBar2 size

Use this feature to set up the VMD config BAR size. The default value is 20.

MemBar2 attribute

Use this feature to set up the VMD config BAR attribute. The options are 32-bit non-prefetchable, 64-bit non-prefetchable, and 64-bit prefetchable.

VMD Config for PStack1

Intel® VMD for Volume Management Device for PStack1

Select Enable to use the Intel Volume Management Device Technology for this stack. The options are Disable and Enable.

*If the feature above is set to Enable, the following features will become available for configuration:

VMD port 2A/2B/2C/2D

Select Enable to use the Intel Volume Management Device Technology for this specific root port. The options are Disable and Enable.

Hot Plug Capable (Available when the device is detected by the system)

Use this feature to enable hot plug support for PCIe root ports 1A\~1D. The options are Disable and Enable.

CfgBar size

Use this feature to set up the VMD config BAR size. The default value is 25.

CfgBar attribute

Use this feature to set up the VMD config BAR attribute. The default option is 64-bit prefetchable.

MemBar1 size

Use this feature to set up the VMD Memory BAR1 size. The default value is 25.

MemBar1 attribute

Use this feature to set up the VMD config BAR attribute. The options are 32-bit non-prefetchable, 64-bit non-prefetchable, and 64-bit prefetchable.

MemBar2 size

Use this feature to set up the VMD config BAR size. The default value is 20.

MemBar2 attribute

Use this feature to set up the VMD config BAR attribute. The options are 32-bit non-prefetchable, 64-bit non-prefetchable, and 64-bit prefetchable.

VMD Config for PStack2

Intel® VMD for Volume Management Device for PStack2

Select Enable to use the Intel Volume Management Device Technology for this stack. The options are Disable and Enable.

*If the feature above is set to Enable, the following features will become available for configuration:

VMD port 3A/3B/3C/3D

Select Enable to use the Intel Volume Management Device Technology for this specific root port. The options are Disable and Enable.

Hot Plug Capable (Available when the device is detected by the system)

Use this feature to enable hot plug support for PCIe root ports 1A\~1D. The options are Disable and Enable.

CfgBar size

Use this feature to set up the VMD config BAR size. The default value is 25.

CfgBar attribute

Use this feature to set up the VMD config BAR attribute. The default option is 64-bit prefetchable.

MemBar1 size

Use this feature to set up the VMD Memory BAR1 size. The default value is 25.

MemBar1 attribute

Use this feature to set up the VMD config BAR attribute. The options are 32-bit non-prefetchable, 64-bit non-prefetchable, and 64-bit prefetchable.

MemBar2 size

Use this feature to set up the VMD config BAR size. The default value is 20.

MemBar2 attribute

Use this feature to set up the VMD config BAR attribute. The options are 32-bit non-prefetchable, 64-bit non-prefetchable, and 64-bit prefetchable.

II0-PCIE Express GGlobal Options

PCI-E Hot Plug

Use this feature to enable hot plug support for PCI-E devices. The options are Disable, Enable, Auto, and Manual.

PCI-E Completion Timeout (Global) Disable

Use this feature to enable PCI-E Completion Timeout support for electric tuning. The options are Yes, No, and Per-Port.

▶South Bridge

The following USB information will display:

• USB Module Version
• USB Devices

Legacy USB Support

This feature enables support for USB 2.0 and older. The options are Enabled, Disabled, and Auto.

XHCI Hand-off

When this feature is disabled, the motherboard will not support USB 3.0. The options are Enabled and Disabled.

Port 60/64 Emulation

This feature allows legacy I/O support for USB devices like mice and keyboards. The options are Enabled and Disabled.

PCIe PLL SSC

Use this feature to enable or disable PCIe PLL SSC. The options are Disabled and Enabled.

Install Window 7 USB Support

Enable this feature to use the USB keyboard and mouse during the Windows 7 installation, since the native XHCI driver support is unavailable. Use a SATA optical drive as a USB drive, and USB CD/DVD drives are not supported. Disable this feature after the XHCI driver has been installed in Windows. The options are Disabled and Enabled.

▶Server ME Configuration

The following General ME Configuration will display:

• Oper. Firmware Version
- Backup Firmware Version
• Recovery Firmware Version
• ME Firmware Status #1
• ME Firmware Status #2
- Current State
- Error Code

▶SATA Configuration

When this submenu is selected, the AMI BIOS automatically detects the presence of the SATA devices that are supported by the Intel PCH chipset and displays the following items:

SATA Controller

Use this feature to enable or disable the onboard SATA controller supported by the Intel PCH chipset. The options are Enable and Disable.

Configure SATA as

Select AHCI to configure a SATA drive specified by the user as an AHCI drive. Select RAID to configure a SATA drive specified by the user as a RAID drive. The options are AHCI and RAID.

SATA HDD Unlock

This feature allows the user to remove any password-protected SATA disk drives. The options are Enable and Disable.

Aggressive Link Power Management

When this feature is enabled, the SATA AHCI controller manages the power usage of the SATA link. The controller will put the link in a low power mode during extended periods of I/O inactivity, and will return the link to an active state when I/O activity resumes. The options are Disabled and Enabled.

SATA Port 0

This item displays the information detected on the installed SATA drive on the particular SATA port.

Hot Plug

Set this feature to Enable for hot plug support, which will allow the user to replace a SATA drive without shutting down the system. The options are Disabled and Enabled.

Spin Up Device

On an edge detect from 0 to 1, set this item to allow the PCH to initialize the device. The options are Disabled and Enabled.

SATA Device Type

Use this feature to specify if the SATA port specified by the user should be connected to a Solid State drive or a Hard Disk Drive. The options are Hard Disk Drive and Solid State Drive.

SATA Port 4

This item displays the information detected on the installed SATA drive on the particular SATA port.

Hot Plug

Set this feature to Enable for hot plug support, which will allow the user to replace a SATA drive without shutting down the system. The options are Disabled and Enabled.

Spin Up Device

On an edge detect from 0 to 1, set this item to allow the PCH to initialize the device. The options are Disabled and Enabled.

SATA Device Type

Use this feature to specify if the SATA port specified by the user should be connected to a Solid State drive or a Hard Disk Drive. The options are Hard Disk Drive and Solid State Drive.

▶sSATA Configuration

When this submenu is selected, the AMI BIOS automatically detects the presence of the SATA devices that are supported by the Intel PCH chipset and displays the following items:

sSATA Controller

Use this feature to enable or disable the onboard SATA controller supported by the Intel PCH chipset. The options are Enable and Disable.

Configure sSATA as

Select AHCI to configure a SATA drive specified by the user as an AHCI drive. Select RAID to configure a SATA drive specified by the user as a RAID drive. The options are AHCI and RAID.

SATA HDD Unlock

This feature allows the user to remove any password-protected SATA disk drives. The options are Disable and Enable.

Aggressive Link Power Management

When this feature is enabled, the SATA AHCI controller manages the power usage of the SATA link. The controller will put the link in a low power mode during extended periods of I/O inactivity, and will return the link to an active state when I/O activity resumes. The options are Disable and Enable.

sSATA Port 0 \~ 5

This item displays the information detected on the installed SATA drive on the particular SATA port.

Hot Plug

Set this feature to Enable for hot plug support, which will allow the user to replace a SATA drive without shutting down the system. The options are Disabled and Enabled.

Spin Up Device

On an edge detect from 0 to 1, set this item to allow the PCH to initialize the device. The options are Disabled and Enabled.

sSATA Device Type

Use this feature to specify if the SATA port specified by the user should be connected to a Solid State drive or a Hard Disk Drive. The options are Hard Disk Drive and Solid State Drive.

▶PCIe/PCI/PnP Configuration

PCI Bus Driver Version

PCI Devices Common Settings

Above 4G Decoding (Available if the system supports 64-bit PCI decoding)

Select Enabled to decode a PCI device that supports 64-bit in the space above 4G Address. The options are Disabled and Enabled.

SR-IOV Support

Use this feature to enable or disable Single Root IO Virtualization support. The options are Disabled and Enabled.

MMIO High Base

Use this feature to select the base memory size according to memory-address mapping for the IO hub. The options are 56T, 48T, 24T, 16T, 4T, and 1T.

MMIO High Granularity Size

Use this feature to select the high memory size according to memory-address mapping for the IO hub. The options are 1G, 4G, 16G, 64G, 256G, and 1024G.

PCI PERR/SERR Support

Use this feature to enable or disable the runtime event for PCI errors. The options are Disabled and Enabled.

Maximum Read Request

Use this feature to select the Maximum Read Request size of the PCI-Express device, or select Auto to allow the system BIOS to determine the value. The options are Auto, 128 Bytes, 256 Bytes, 512 Bytes, 1024 Bytes, 2048 Bytes, and 4096 Bytes.

MMCFG Base

Use this feature to select the low base address for PCI-E adapters to increase base memory. The options are 1G, 1.5G, 1.75G, 2G, 2.25G, and 3G.

VGA Priority

Use this feature to select VGA priority when multiple VGA devices are detected. Select On-board to give priority to your onboard video device. Select Offboard to give priority to your graphics card. The options are Auto, Onboard, and Offboard.

PCIe/PCI/PnP Configuration

Onboard NVME 1 OPROM

Use this feature to select which firmware to be loaded for the add-on card in this slot. The options are Disabled, Legacy, and EFI.

Onboard NVME 2 OPROM

Use this feature to select which firmware to be loaded for the add-on card in this slot. The options are Disabled, Legacy, and EFI.

CPU MICRO-LP PCI-E 3.0 X8

Use this feature to select which firmware to be loaded for the add-on card in this slot. The options are Disabled, Legacy, and EFI.

CPU SLOT PCI-E 3.0 X8

Use this feature to select which firmware to be loaded for the add-on card in this slot. The options are Disabled, Legacy, and EFI.

Onboard Video OPROM

Use this feature to select the Onboard Video Option ROM type. The options are Disabled, Legacy, and UEFI.

▶Network Stack Configuration

Network Stack

Select Enabled to enable PXE (Preboot Execution Environment) or UEFI (Unified Extensible Firmware Interface) for network stack support. The options are Disabled and Enabled.

*If the feature above is set to Enabled, the next seven features will be available for configuration:

Ipv4 PXE Support

Select Enabled to enable IPv4 PXE boot support. The options are Disabled and Enabled.

Ipv4 HTTP Support

Select Enabled to enable IPv4 HTTP boot support. The options are Disabled and Enabled.

Ipv6 PXE Support

Select Enabled to enable IPv6 PXE boot support. The options are Disabled and Enabled.

Ipv6 HTTP Support

Select Enabled to enable IPv6 HTTP boot support. The options are Disabled and Enabled.

IPSEC Certificate

The feature appears if Network Stack is enabled. Internet Protocol Security (IPSEC) offers a secure connection for remote computers using a secure tunnel. The options are Disabled and Enabled.

PXE boot wait time

Use this feature to specify the wait time to press the ESC key to abort the PXE boot. Press "+" or "-" on your keyboard to change the value. The default setting is 0.

Media detect count

Use this option to specify the number of times media will be checked. Press "+" or "-" on your keyboard to change the value. The default setting is 1.

▶ Super IO Configuration

Super IO Configuration

Super IO Chip AST2500

▶ Serial Port 1 Configuration

Serial Port

Select Enabled to enable the selected onboard serial port. The options are Disabled and Enabled.

Device Settings

This feature displays the status of a serial port specified by the user.

Change Settings

This feature specifies the base I/O port address and the Interrupt Request address of a serial port specified by the user. Select Auto to allow the BIOS to automatically assign the base I/O and IRQ address. The options are Auto, (IO=3F8h; IRQ=4); (IO=3F8h; IRQ=3, 4, 5, 6, 7, 9, 10, 11, 12); (IO=2F8h; IRQ=3, 4, 5, 6, 7, 9, 10, 11, 12), (IO=3E8h; IRQ=3, 4, 5, 6, 7, 9, 10, 11, 12), and (O=2E8h; IRQ=3, 4, 5, 6, 7, 9, 10, 11, 12).

▶ Serial Port 2 Configuration

Serial Port

Select Enabled to enable the selected onboard serial port. The options are Disabled and Enabled.

Device Settings

This feature displays the status of a serial port specified by the user.

Change Settings

This feature specifies the base I/O port address and the Interrupt Request address of a serial port specified by the user. Select Auto to allow the BIOS to automatically assign the base I/O and IRQ address. The options are Auto, (IO=2F8h; IRQ=3); (IO=3F8h; IRQ=3, 4, 5, 6, 7, 9, 10, 11, 12); (IO=2F8h; IRQ=3, 4, 5, 6, 7, 9, 10, 11, 12), (IO=3E8h; IRQ=3, 4, 5, 6, 7, 9, 10, 11, 12), and (O=2E8h; IRQ=3, 4, 5, 6, 7, 9, 10, 11, 12).

Serial Port 2 Attribute

Use this feature to select the serial port 2 mode. The options are SOL and COM.

▶ Serial Port Console Redirection

COM1

Console Redirection

Select Enabled to enable COM1 for Console Redirection, which will allow a client machine to be connected to a host machine at a remote site for networking. The options are Disabled and Enabled.

*If the feature above is set to Enabled, the following features will become available for configuration:

▶Console Redirection Settings

COM1

Console Redirection Settings

Terminal Type

This feature allows the user to select the target terminal emulation type for Console Redirection. Select VT100 to use the ASCII Character set. Select VT100+ to add color and function key support. Select ANSI to use the Extended ASCII Character Set. Select VT-UTF8 to use UTF8 encoding to map Unicode characters into one or more bytes. The options are VT100, VT100+, VT-UTF8, and ANSI.

Bits per second

Use this feature to set the transmission speed for a serial port used in Console Redirection. Make sure that the same speed is used in the host computer and the client computer. A lower transmission speed may be required for long and busy lines. The options are 9600, 19200, 38400, 57600, and 115200 (bits per second).

Data Bits

Use this feature to set the data transmission size for Console Redirection. The options are 7 and 8.

Parity

A parity bit can be sent along with regular data bits to detect data transmission errors. Select Even if the parity bit is set to 0, and the number of 1's in data bits is even. Select Odd if the parity bit is set to 0, and the number of 1's in data bits is odd. Select None if you do not want to send a parity bit with your data bits in transmission. Select Mark to add a mark as a parity bit to be sent along with the data bits. Select Space to add a Space as a parity bit to be sent with your data bits. The options are None, Even, Odd, Mark, and Space.

Stop Bits

A stop bit indicates the end of a serial data packet. Select 1 Stop Bit for standard serial data communication. Select 2 Stop Bits if slower devices are used. The options are 1 and 2.

Flow Control

Use this feature to set the flow control for Console Redirection to prevent data loss caused by buffer overflow. Send a "Stop" signal to stop sending data when the receiving buffer is full. Send a "Start" signal to start sending data when the receiving buffer is empty. The options are None and Hardware RTS/CTS.

VT-UTF8 Combo Key Support

Select Enabled to enable VT-UTF8 Combination Key support for ANSI/VT100 terminals. The options are Disabled and Enabled.

Recorder Mode

Select Enabled to capture the data displayed on a terminal and send it as text messages to a remote server. The options are Disabled and Enabled.

Resolution 100x31

Select Enabled for extended-terminal resolution support. The options are Disabled and Enabled.

Putty KeyPad

This feature selects Function Keys and KeyPad settings for Putty, which is a terminal emulator designed for the Windows OS. The options are VT100, LINUX, XTERMR6, SCO, ESCN, and VT400.

COM2/SOL

Console Redirection

Select Enabled to enable COM1 for Console Redirection, which will allow a client machine to be connected to a host machine at a remote site for networking. The options are Disabled and Enabled.

*If the feature above is set to Enabled, the following features will become available for configuration:

▶Console Redirection Settings

COM2/SOL

Console Redirection Settings

Terminal Type

This feature allows the user to select the target terminal emulation type for Console Redirection. Select VT100 to use the ASCII Character set. Select VT100+ to add color

and function key support. Select ANSI to use the Extended ASCII Character Set. Select VT-UTF8 to use UTF8 encoding to map Unicode characters into one or more bytes. The options are VT100, VT100+, VT-UTF8, and ANSI.

Bits per second

Use this feature to set the transmission speed for a serial port used in Console Redirection. Make sure that the same speed is used in the host computer and the client computer. A lower transmission speed may be required for long and busy lines. The options are 9600, 19200, 38400, 57600, and 115200 (bits per second).

Data Bits

Use this feature to set the data transmission size for Console Redirection. The options are 7 and 8.

Parity

A parity bit can be sent along with regular data bits to detect data transmission errors. Select Even if the parity bit is set to 0, and the number of 1's in data bits is even. Select Odd if the parity bit is set to 0, and the number of 1's in data bits is odd. Select None if you do not want to send a parity bit with your data bits in transmission. Select Mark to add a mark as a parity bit to be sent along with the data bits. Select Space to add a Space as a parity bit to be sent with your data bits. The options are None, Even, Odd, Mark, and Space.

Stop Bits

A stop bit indicates the end of a serial data packet. Select 1 Stop Bit for standard serial data communication. Select 2 Stop Bits if slower devices are used. The options are 1 and 2.

Flow Control

Use this feature to set the flow control for Console Redirection to prevent data loss caused by buffer overflow. Send a "Stop" signal to stop sending data when the receiving buffer is full. Send a "Start" signal to start sending data when the receiving buffer is empty. The options are None and Hardware RTS/CTS.

VT-UTF8 Combo Key Support

Select Enabled to enable VT-UTF8 Combination Key support for ANSI/VT100 terminals. The options are Disabled and Enabled.

Recorder Mode

Select Enabled to capture the data displayed on a terminal and send it as text messages to a remote server. The options are Disabled and Enabled.

Resolution 100x31

Select Enabled for extended-terminal resolution support. The options are Disabled and Enabled.

Putty KeyPad

This feature selects Function Keys and KeyPad settings for Putty, which is a terminal emulator designed for the Windows OS. The options are VT100, LINUX, XTERMR6, SCO, ESCN, and VT400.

Legacy Console Redirection

▶Legacy Console Redirection Settings

Legacy Redirection COM Port

Use this feature to select a COM port to display redirection of Legacy OS and Legacy OPROM messages. The options are COM1 and COM2/SOL.

Legacy OS Redirection Resolution

Use this feature to select the number of rows and columns used in Console Redirection for legacy OS support. The options are 80x24 and 80x25.

Redirection After BIOS POST

Use this feature to enable or disable legacy console redirection after BIOS POST. When set to BootLoader, legacy console redirection is disable before booting the OS. When set to Always Enable, legacy console redirection remains enabled when booting the OS. The options are Always Enable and BootLoader.

Serial Port for Out-of-Band Management/Windows Emergency Management Services (EMS)

Console Redirection

Select Enabled to use a COM port selected by the user for EMS Console Redirection. The options are Disabled and Enabled.

*If the feature above is set to Enabled, the following features are available for configuration:

▶Console Redirection Settings

This feature allows the user to specify how the host computer will exchange data with the client computer, which is the remote computer used by the user.

Out-of-Band Mgmt Port

The feature selects a serial port in a client server to be used by the Microsoft Windows Emergency Management Services (EMS) to communicate with a remote host server. The options are COM1 and COM2/SOL.

Terminal Type

Use this feature to select the target terminal emulation type for Console Redirection. Select VT100 to use the ASCII character set. Select VT100+ to add color and function key support. Select ANSI to use the extended ASCII character set. Select VT-UTF8 to use UTF8 encoding to map Unicode characters into one or more bytes. The options are VT100, VT100+, VT-UTF8, and ANSI.

Bits per second

This feature sets the transmission speed for a serial port used in Console Redirection. Make sure that the same speed is used in the host computer and the client computer. A lower transmission speed may be required for long and busy lines. The options are 9600, 19200, 57600, and 115200 (bits per second).

Flow Control

Use this feature to set the flow control for Console Redirection to prevent data loss caused by buffer overflow. Send a "Stop" signal to stop sending data when the receiving buffer is full. Send a "Start" signal to start sending data when the receiving buffer is empty. The options are None, Hardware RTS/CTS, and Software Xon/Xoff.

Data Bits

Parity

Stop Bits

▶ACPI Settings

NUMA Support (Available when the OS supports this feature)

Select Enabled to enable Non-Uniform Memory Access support to enhance system performance. The options are Disabled and Enabled.

WHEA Support

This feature Enables the Windows Hardware Error Architecture (WHEA) support for the Windows 2008 (or a later version) operating system. The options are Disabled and Enabled.

High Precision Event Timer

Select Enabled to activate the High Performance Event Timer (HPET) that produces periodic interrupts at a much higher frequency than a Real-time Clock (RTC) does in synchronizing multimedia streams, providing smooth playback and reducing the dependency on other timestamp calculation devices, such as an x86 RDTSC Instruction embedded in the CPU. The High Performance Event Timer is used to replace the 8254 Programmable Interval Timer. The options are Disabled and Enabled.

▶Trusted Computing

*This motherboard supports TPM 1.2 and 2.0. The following Trusted Platform Module (TPM) information will display if a TPM 2.0 module is detected:

• Vendor
• Firmware Version

Security Device Support

If this feature and the TPM jumper on the motherboard are both set to Enabled, onboard security devices will be enabled for TPM support to enhance data integrity and network security. Please reboot the system for a change on this setting to take effect. The options are Disable and Enable.

• Active PCR Bank
• Available PCR Bank

*If the feature "Security Device Support" is enabled, the following features are available for configuration:

SHA-1 PCR Bank

Use this feature to disable or enable the SHA-1 Platform Configuration Register (PCR) bank for the installed TPM device. The options are Disabled and Enabled.

SHA256 PCR Bank

Use this feature to disable or enable the SHA256 Platform Configuration Register (PCR) bank for the installed TPM device. The options are Disabled and Enabled.

Pending operation

Use this feature to schedule a TPM-related operation to be performed by a security device for system data integrity. Your system will reboot to carry out a pending TPM operation. The options are None and TPM Clear.

Platform Hierarchy

Use this feature to disable or enable platform hierarchy for platform protection. The options are Disabled and Enabled.

Storage Hierarchy

Use this feature to disable or enable storage hierarchy for cryptographic protection. The options are Disabled and Enabled.

Endorsement Hierarchy

Use this feature to disable or enable endorsement hierarchy for privacy control. The options are Disabled and Enabled.

PH Randomization

Use this feature to disable or enable Platform Hierarchy (PH) Randomization. The options are Disabled and Enabled.

TXT Support

Intel Trusted Execution Technology (TXT) helps protect against software-based attacks and ensures protection, confidentiality, and integrity of data stored or created on the system. Use this feature to enable or disable TXT Support. The options are Disabled and Enabled.

▶TLS Authentication Configuration

This submenu allows the user to configure Transport Layer Security (TLS) settings.

▶Server CA Configuration

▶Enroll Cert

Enroll Cert Using File

Use this feature to enroll certification from a file.

Cert GUID

Use this feature to input the certification GUID.

▶ Commit Changes and Exit

Use this feature to save all changes and exit TLS settings.

▶Discard Changes and Exit

Use this feature to discard all changes and exit TLS settings.

▶ Delete Certification

Use this feature to delete certification.

▶Client Cert Configuration

▶Enroll Cert

Enroll Cert Using File

Use this feature to enroll certification from a file.

Cert GUID

Use this feature to input the certification GUID.

▶ Commit Changes and Exit

Use this feature to save all changes and exit TLS settings.

▶Discard Changes and Exit

Use this feature to discard all changes and exit TLS settings.

▶ Delete Certification

Use this feature to delete certification.

▶Intel® Virtual RAID on CPU

This submenu displays RAID volumes, if detected by the system.

▶All Cpu Information

The following CPU information will display:

• Total CPU Number
• Backup Firmware Version
- CPU0 - CPU15
- CPUID
- Stepping
- MicroCodeRev
- PlatformID
- CpuCoreFreq (MHz)
• ActualCoreFreq (MHz)

6.4 IPMI

Use this tab page to configure Intelligent Platform Management Interface (IPMI) settings.

Aptio Setup Utility - Copyright (C) 2018 American Megatrends, Inc. Main Advanced IPMI Security Event Logs Boot Save & Exit IPMI Firmware Revision 9.51 Status Of BMC Working ► System Event Log ► BMC Network Configuration Press to change the SEL event log configuration. ++: Select Screen ↑↓: Select Item Enter: Select +/-: Change Opt. F1: General Help F2: Previous Values F3: Optimized Defaults F4: Save & Exit ESC: Exit Version 2.20.1271. Copyright (C) 2018 American Megatrends, Inc.

The following items will be displayed:

• IPMI Firmware Revision
- Status of BMC

▶System Event Log

Enabling/Disabling Options

SEL Components

Select Enabled for all system event logging at boot up. The options are Disabled and Enabled.

Erasing Settings

Erase SEL

Select Yes, On next reset to erase all system event logs upon next system reboot. Select Yes, On every reset to erase all system event logs upon each system reboot. Select No

to keep all system event logs after each system reboot. The options are No, Yes, On next reset, and Yes, On every reset.

When SEL is Full

This feature allows the user to determine what the BIOS should do when the system event log is full. Select Erase Immediately to erase all events in the log when the system event log is full. The options are Do Nothing and Erase Immediately.

Note: After making changes on a setting, reboot the system for the changes to take effect.

▶BMC Network Configuration

IPMI LAN Selection

IPMI Network Link Status

Station MAC Address

VLAN

IPv4 Address Source

Station IP Address

Subnet Mask

Gateway IP address

IPV6 address status

Station IPV6 address

Prefix Length

IPV6 Router IP Address

Update IPMI LAN Configuration

Select Yes for the BIOS to implement all IP/MAC address changes at the next system boot. The options are No and Yes.

*If the feature above is set to Yes, the Configuration Address Source and VLAN features are available for configuration:

IPMI LAN Selection

This feature displays the IPMI LAN setting. The options are Dedicated, Shared, and Failover.

VLAN

This feature is configurable if the Update IPMI LAN Configuration feature is set to Yes. Use this feature to enable or disable the IPMI VLAN function. The options are Disable and Enable.

*If the feature above is set to Enable, the VLAN feature below is available for configuration:

VLAN

Use this feature to select a value for VLAN ID.

Configure IPV4 support

Configuration Address Source

Use this feature to select the source of the IP address for this computer. If Static is selected, you will need to know the IP address of this computer and enter it to the system manually in the field. If DHCP is selected, the BIOS will search for a DHCP (Dynamic Host Configuration Protocol) server in the network that it is attached to and request the next available IP address for this computer. The options are DHCP and Static.

*If the feature above is set to Static, the Station IP Address/Subnet Mask/Gateway IP Address features are available for configuration:

Station IP Address

Use this feature to set the IP address.

Subnet mask

Use this feature to set the subnet mask address.

Gateway IP Address

Use this feature to set the Gateway IP address.

Configure IPV6 support

IPV6 Support

Use this feature to enable or disable IPV6 support for LAN1. The options are Enabled and Disabled.

6.5 Security

Use this tab page to configure Security settings.

Aptio Setup Utility - Copyright (C) 2018 American Megatrends, Inc. Main Advanced IPMI Security Event Logs Boot Save & Exit
Password Description If ONLY the Administrator's password is set, then this only limits access to Setup and is only asked for when entering Setup. If ONLY the User's password is set, then this is a power on password and must be entered to boot or enter Setup. In Setup the User will have Administrator rights. The password length must be in the following range: Minimum length 3 Maximum length 20 Administrator Password User Password Password Check [Setup] Secure Boot	Set Administrator Password
	+: Select Screen ↑↓: Select Item Enter: Select +/-: Change Opt. F1: General Help F2: Previous Values F3: Optimized Defaults F4: Save & Exit ESC: Exit
Version 2.20.1271. Copyright (C) 2018 American Megatrends, Inc.

Administrator Password

Use this feature to set the administrator password which is required to enter the BIOS setup utility. The length of the password should be from 3 to 20 characters long.

User Password

Use this feature to set a user password.

Password Check

Select Setup for the system to check for a password at Setup. Select Always for the system to check for a password at bootup or upon entering the BIOS Setup utility. The options are Setup and Always.

▶ Secure Boot

• System Mode
• Secure Boot

Secure Boot

Select Enable for secure boot support to ensure system security at boot up. The options are Disabled and Enabled.

Secure Boot Mode

This feature allows the user to select the desired secure boot mode for the system. The options are Standard and Custom.

*If Secure Boot Mode is set to Customized, Key Management features are available for configuration:

CSM Support

This feature is for manufacturing debugging purposes.

Vendor Keys

Select Yes to restore all factory keys to the default settings. The options are Yes and No.

▶ Reset to Setup Mode

Select Yes to delete all Secure Boot key databases and force the system to Setup Mode. The options are Yes and No.

▶Key Management

Factory Key Provision

Select Enabled to install the default Secure Boot keys set by the manufacturer. The options are Disabled and Enabled.

Select Yes to restore all factory keys to the default settings. The options are Yes and No.

▶ Reset to Setup Mode

Select Yes to delete all Secure Boot key databases and force the system to Setup Mode. The options are Yes and No.

▶ Export Secure Boot variables

Use this feature to copy the NVRAM contents of the secure boot variables to a file.

▶Enroll Efi Image

This feature allows the image to run in Secure Boot mode.

Device Guard Ready

▶Remove 'UEFI CA' from DB

Use this feature to remove the Microsoft UEFI CA certificate from the database. The options are Yes and No.

Select Yes to restore all DBs to the default settings. The options are Yes and No.

Secure Boot variable

▶ Platform Key (PK)

Details

Select this feature to view the details of the Platform Key.

Export

Select Yes to export a PK from a file on an external media.

Update

Select Yes to load a factory default PK or No to load from a file on an external media.

Delete

Select Ok to remove the PK and then the system will reset to Setup/Audit Mode.

▶Key Exchange Keys

Details

Select this feature to view the details of the Key Exchange Key.

Export

Select Yes to export a KEK from a file on an external media.

Update

Select Yes to load a factory default KEK or No to load from a file on an external media.

Append

Select Yes to add the KEK from the manufacturer's defaults list to the existing KEK. Select No to load the KEK from a file. The options are Yes and No.

Delete

Select Ok to remove the KEK and then the system will reset to Setup/Audit Mode.

▶ Authorized Signatures

Details

Select this feature to view the details of the db.

Export

Select Yes to export a db from a file on an external media.

Update

Select Yes to load a factory default db or No to load from a file on an external media.

Append

Select Yes to add the db from the manufacturer's defaults list to the existing db. Select No to load the db from a file. The options are Yes and No.

Delete

Select Ok to remove the db and then the system will reset to Setup/Audit Mode.

▶Forbidden Signatures

Details

Select this feature to view the details of the dbx.

Export

Select Yes to export a dbx from a file on an external media.

Update

Select Yes to load a factory default dbx or No to load from a file on an external media.

Append

Select Yes to add the dbx from the manufacturer's defaults list to the existing dbx. Select No to load the dbx from a file. The options are Yes and No.

Delete

Select Ok to remove the dbx and then the system will reset to Setup/Audit Mode.

▶ Authorized TimeStamps

Update

Select Yes to load a factory default dbt or No to load from a file on an external media.

Append

Select Yes to add the dbt from the manufacturer's defaults list to the existing dbt. Select No to load the dbt from a file. The options are Yes and No.

▶OsRecovery Signatures

Update

Select Yes to load a factory default dbr or No to load from a file on an external media.

Append

Select Yes to add the dbr from the manufacturer's defaults list to the existing dbr. Select No to load the dbr from a file. The options are Yes and No.

6.6 Event Logs

Use this tab page to configure Event Log settings.

Aptio Setup Utility - Copyright (C) 2018 American Megatrends, Inc. Main Advanced IPMI Security Event Logs Boot Save & Exit Change Smbios Event Log Settings View Smbios Event Log Press to change the Smbios Event Log configuration. +: Select Screen ↑↓: Select Item Enter: Select +/-: Change Opt. F1: General Help F2: Previous Values F3: Optimized Defaults F4: Save & Exit ESC: Exit Version 2.20.1271. Copyright (C) 2018 American Megatrends, Inc.

▶Change SMBIOS Event Log Settings

Enabling/Disabling Options

Smbios Event Log

Change this feature to enable or disable all features of the SMBIOS Event Logging during system boot. The options are Disabled and Enabled.

Erasing Settings

Erase Event Log

Select Enabled to erase all error events in the SMBIOS (System Management BIOS) log before an event logging is initialized at bootup. The options are No, Yes, Next reset, and Yes, Every reset.

When Log is Full

Select Erase Immediately to immediately erase all errors in the SMBIOS event log when the event log is full. Select Do Nothing for the system to do nothing when the SMBIOS event log is full. The options are Do Nothing and Erase Immediately.

Smbios Event Log Standard Settings

Log System Boot Event

Select Enabled to log system boot events. The options are Enabled and Disabled.

MECI (Multiple Event Count Increment)

Enter the increment value for the multiple event counter. Enter a number between 1 to 255. The default setting is 1.

METW (Multiple Event Count Time Window)

This feature is used to determine how long (in minutes) should the multiple event counter wait before generating a new event log. Enter a number between 0 to 99. The default setting is 60.

Note: Please reboot the system for the changes to take effect.

▶View Smbios Event Log

This feature allows the user to view the event in the SMBIOS event log. The following categories are displayed:

DATE/TIME/ERROR CODE/SEVERITY

6.7 Boot

Use this tab page to configure Boot Settings.

Aptio Setup Utility - Copyright (C) 2018 American Megatrends, Inc. Main Advanced IPMI Security Event Logs Boot Save & Exit Boot mode select [OUAL] FIXED BOOT ORDER Priorities Boot Option #1 [Hard Disk] Boot Option #2 [CD/DVD] Boot Option #3 [USB Hard Disk] Boot Option #4 [USB CD/DVD] Boot Option #5 [USB Key] Boot Option #6 [USB Floppy] Boot Option #7 [USB Lan] Boot Option #8 [Network] Boot Option #9 [UEFI Hard Disk] Boot Option #10 [UEFI CD/DVD] Boot Option #11 [UEFI USB Hard Disk] Boot Option #12 [UEFI USB CD/DVD] Boot Option #13 [UEFI USB Key] Boot Option #14 [UEFI USB Floppy] Boot Option #15 [UEFI USB Lan] Boot Option #16 [UEFI Network] Boot Option #17 [UEFI AP:UEFI: Built-in EFI Shell] ► Delete Boot Option ▲ Select boot mode LEGACY/UEFI ++: Select Screen ↑↓: Select Item Enter: Select +/-: Change Opt. F1: General Help F2: Previous Values F3: Optimized Defaults F4: Save & Exit ESC: Exit Version 2.20.1271. Copyright (C) 2018 American Megatrends, Inc.

Setup Prompt Timeout

Use this feature to specify the length of time (the number of seconds) for the BIOS to wait before rebooting the system when the setup activation key is pressed. Enter the value of 65535 (0xFFFF) for the BIOS to wait indefinitely. The default setting is 1.

Boot mode select

Use this feature to select the boot mode. The options are LEGACY, UEFI, and DUAL.

Fixed BOOT ORDER Priorities

This option prioritizes the order of bootable devices that the system can boot from. Press on each entry from top to bottom to select devices.

• LEGACY/UEFI/DUAL Boot Order #1
• LEGACY/UEFI/DUAL Boot Order #2
• LEGACY/UEFI/DUAL Boot Order #3
• LEGACY/UEFI/DUAL Boot Order #4
• LEGACY/UEFI/DUAL Boot Order #5
• LEGACY/UEFI/DUAL Boot Order #6
• LEGACY/UEFI/DUAL Boot Order #7
• LEGACY/UEFI/DUAL Boot Order #8
• LEGACY/UEFI/DUAL Boot Order #9
• LEGACY/UEFI/DUAL Boot Order #10
• LEGACY/UEFI/DUAL Boot Order #11
• LEGACY/UEFI/DUAL Boot Order #12
• LEGACY/UEFI/DUAL Boot Order #13
• LEGACY/UEFI/DUAL Boot Order #14
• LEGACY/UEFI/DUAL Boot Order #15
• LEGACY/UEFI/DUAL Boot Order #16
• LEGACY/UEFI/DUAL Boot Order #17

▶ Delete Boot Option

Use this feature to select a boot device to delete from the boot priority list.

▶ Delete Driver Option

Use this feature to remove an EFI driver option from the driver order.

▶UEFI Application Boot Priorities

- Boot Option # - This feature sets the system boot order of detected devices. The options are [the list of detected boot device(s)] and Disabled.

6.8 Save & Exit

Use this tab page to configure Save & Exit settings.

Aptio Setup Utility - Copyright (C) 2018 American Megatrends, Inc. Main Advanced IPMI Security Event Logs Boot Save & Exit Save Options Discard Changes and Exit Save Changes and Reset Save Changes Discard Changes Default Options Restore Optimized Defaults Save as User Defaults Restore User Defaults Boot Override UEFI: Built-in EFI Shell Launch EFI Shell from filesystem device Exit system setup without saving any changes. ++: Select Screen ↑↓: Select Item Enter: Select +/-: Change Opt. F1: General Help F2: Previous Values F3: Optimized Defaults F4: Save & Exit ESC: Exit Version 2.20.1271. Copyright (C) 2018 American Megatrends, Inc.

Save Options

Discard Changes and Exit

Select this feature to quit the BIOS Setup without making any permanent changes to the system configuration and reboot the computer. Select Discard Changes and Exit from the Exit menu and press .

Save Changes and Reset

When you have completed the system configuration changes, select this option to save all changes made and reset the system.

Save Changes

When you have completed the system configuration changes, select this option to save all changes made. This will not reset (reboot) the system.

Discard Changes

Select this option and press to discard all the changes and return to the AMI BIOS Utility Program.

Default Options

Restore Defaults

To set this feature, select Restore Optimized Defaults and press . These are factory settings designed for maximum system performance but not for maximum stability.

Save as User Defaults

To set this feature, select Save as User Defaults from the Exit menu and press . This enables the user to save any changes to the BIOS setup for future use.

To set this feature, select Restore User Defaults from the Exit menu and press . Use this feature to retrieve user-defined settings that were saved previously.

Boot Override

Other boot options are listed in this section. The system will boot to the selected boot option.

UEFI: Built-in EFI Shell

Launch EFI Shell from filesystem device

Appendix A

BIOS Error Codes

A.1 BIOS Error Beep (POST) Codes

During the POST (Power-On Self-Test) routines, which are performed each time the system is powered on, errors may occur.

Non-fatal errors are those which, in most cases, allow the system to continue the boot-up process. The error messages normally appear on the screen.

Fatal errors are those which will not allow the system to continue the boot-up procedure. If a fatal error occurs, you should consult with your system manufacturer for possible repairs.

These fatal errors are usually communicated through a series of audible beeps. The table below lists some common errors and their corresponding beep codes encountered by users.

BIOS Error Beep (POST) Codes
Beep Code Error	Message Description
1 short Refresh Circuits have been reset (Ready to power up)
5 short, 1 long Memory error No memory detected in system
5 long, 2 short Display memory read/write error Video adapter missing or with faulty memory
1 long continuous System OH System overheat condition

A.2 Additional BIOS POST Codes

The AMI BIOS supplies additional checkpoint codes, which are documented online at http://www.supermicro.com/support/manuals/ ("AMI BIOS POST Codes User's Guide").

When BIOS performs the Power On Self Test, it writes checkpoint codes to I/O port 0080h. If the computer cannot complete the boot process, a diagnostic card can be attached to the computer to read I/O port 0080h (Supermicro p/n AOC-LPC80-20).

For information on AMI updates, please refer to http://www.ami.com/products/.

Appendix B

Standardized Warning Statements for AC Systems

About Standardized Warning Statements

The following statements are industry standard warnings, provided to warn the user of situations which have the potential for bodily injury. Should you have questions or experience difficulty, contact Supermicro's Technical Support department for assistance. Only certified technicians should attempt to install or configure components.

Read this appendix in its entirety before installing or configuring components in the Supermicro chassis.

These warnings may also be found on our website at http://www.supermicro.com/about/policies/safety_information.cfm.

Warning Definition

Warning! This warning symbol means danger. You are in a situation that could cause bodily injury. Before you work on any equipment, be aware of the hazards involved with electrical circuitry and be familiar with standard practices for preventing accidents.

警告の定義

この警告サインは危険を意味します。

Installation Instructions

Warning! Read the installation instructions before connecting the system to the power source.

設置手順書

Warning! This product relies on the building's installation for short-circuit (overcurrent) protection. Ensure that the protective device is rated not greater than: 250 V, 20 A.

サーキット・ブレーカー

Power Disconnection Warning

Warning! The system must be disconnected from all sources of power and the power cord removed from the power supply module(s) before accessing the chassis interior to install or remove system components.

電源切断の警告

Equipment Installation

Warning! Only trained and qualified personnel should be allowed to install, replace, or service this equipment.

機器の設置

Warning! This unit is intended for installation in restricted access areas. A restricted access area can be accessed only through the use of a special tool, lock and key, or other means of security. (This warning does not apply to workstations).

アクセス制限区域

Warning! There is the danger of explosion if the battery is replaced incorrectly. Replace the battery only with the same or equivalent type recommended by the manufacturer. Dispose of used batteries according to the manufacturer's instructions

電池の取り扱い

Redundant Power Supplies

Warning! This unit might have more than one power supply connection. All connections must be removed to de-energize the unit.

冗長電源装置

Warning! Hazardous voltage or energy is present on the backplane when the system is operating. Use caution when servicing.

バックプレーンの電圧

Comply with Local and National Electrical Codes

Warning! Installation of the equipment must comply with local and national electrical codes.

地方および国の電気規格に準拠

Warning! Ultimate disposal of this product should be handled according to all national laws and regulations.

製品の廃棄

Warning! Hazardous moving parts. Keep away from moving fan blades. The fans might still be turning when you remove the fan assembly from the chassis. Keep fingers, screwdrivers, and other objects away from the openings in the fan assembly's housing.

ファン・ホットスワップの警告

Power Cable and AC Adapter

Warning! When installing the product, use the provided or designated connection cables, power cables and AC adaptors. Using any other cables and adaptors could cause a malfunction or a fire. Electrical Appliance and Material Safety Law prohibits the use of UL or CSA-certified cables (that have UL/CSA shown on the cord) for any other electrical devices than products designated by Supermicro only.

電源コードとACアダプター

System Specifications

Processors (per node)

For 5039MD8-H8TNR single Intel Xeon D-2141I processor, 8 Core, 16 Threads, 65W, System on Chip For 5039MD18-H8TNR single Intel Xeon D-2191 processor, 18 Core, 36 Threads, 86W, System on Chip Note: Please refer to the motherboard specifications pages on our website for updates to supported processors.

BIOS

128Mb SPI Flash EEPROM with AMI BIO

Memory (per node)

Up to 512GB ECC LRDIMM or 256GB ECC RDIMM in four DDR4 slots with speed of up to 2400MHz

Storage Drives

Sixteen (two per node) 3.5" SATA3 by default; optional kits allow sixteen 2.5" hybrid SATA3/NVMe M.2: Up to two PCI-E M.2 SSDs

PCI Expansion Slots (per node)

One low-profile PCI-E slot using a riser card

Input/Output (per node)

LAN: Two Gb Ethernet ports; one dedicated IPMI port via Micro-LP network card With KVM dongle: two USB 2.0 ports, one VGA port; one COM port (per system) 1+1 dedicated IPMI LAN port for 8 nodes management

Motherboard (per node)

X11SDD-8C-F for 5039MD8-H8TNR; X11SDD-18C-F for 5039MD18-H8TNR; 4.66" x 18.50" (118.4 mm x 469.9 mm)

Chassis

SC938NH-R1K68B; 3U rackmount; (WxHxD) 15.26 x 5.2 x 23 in. (387 x 132 x 585 mm)

System Cooling

Four 80x80x38 mm, 10.5K RPM fans, CPU heatsinks, air shrouds to direct air flow

Weight

Net Weight: 62.2 lbs (28.21 kg) Gross Weight: 88 lbs (39.92 kg)

Power Supply

Model: PWS-1K68A-1R, Dual 1600 W redundant 80Plus Titanium level modules Input: 100-140Vac / 10-7A / 50-60Hz 180-240Vac / 11-8A / 50-60Hz +12V Max: 67A / Min: 0A (100-140Vac) Max: 133A / Min: 0A (180-240Vac) +5Vsb Max: 1A / Min: 0A

Operating Environment

Operating Temperature: 10^ to 35^ C ( 50^ to 95^ F) Non-operating Temperature: -40^ to 70^ C ( -40^ to 158^ F) Operating Relative Humidity: 8% to 90% (non-condensing) Non-operating Relative Humidity: 5% to 95% (non-condensing)

Regulatory Compliance

Electromagnetic Emissions: FCC Class A, EN 55032 Class A, EN 61000-3-2/3-3, CISPR 32 Class A

Electromagnetic Immunity: EN 55024/CISPR 24, (EN 61000-4-2, EN 61000-4-3, EN 61000-4-4, EN 61000-4-5, EN 61000-4-6, EN 61000-4-8, EN 61000-4-11)

Other: VCCI-CISPR 32 and AS/NZS CISPR 32

Environmental: Directive 2011/65/EU and Delegated Directive (EU) 2015/863 and Directive 2012/19/EU

Safety: CSA/EN/IEC/UL 60950-1 Compliant, UL or CSA Listed (USA and Canada), CE Marking (Europe)

Perchlorate Warning

California Best Management Practices Regulations for Perchlorate Materials: This Perchlorate warning applies only to products containing CR (Manganese Dioxide) Lithium coin cells. "Perchlorate Material-special handling may apply. See www.dtsc.ca.gov/hazardouswaste/perchlorate"

Appendix D

UEFI BIOS Recovery

Warning: Do not upgrade the BIOS unless your system has a BIOS-related issue. Flashing the wrong BIOS can cause irreparable damage to the system. In no event shall Supermicro be liable for direct, indirect, special, incidental, or consequential damages arising from a BIOS update. If you do update the BIOS, do not shut down or reset the system while the BIOS is updating to avoid possible boot failure.

D.1 Overview

The Unified Extensible Firmware Interface (UEFI) provides a software-based interface between the operating system and the platform firmware in the pre-boot environment. The UEFI specification supports an architecture-independent mechanism that will allow the UEFI OS loader stored in an add-on card to boot the system. The UEFI offers clean, hands-off management to a computer during system boot.

D.2 Recovering the UEFI BIOS Image

A UEFI BIOS flash chip consists of a recovery BIOS block and a main BIOS block (a main BIOS image). The recovery block contains critical BIOS codes, including memory detection and recovery codes for the user to flash a healthy BIOS image if the original main BIOS image is corrupted. When the system power is turned on, the recovery block codes execute first. Once this process is complete, the main BIOS code will continue with system initialization and the remaining POST (Power-On Self-Test) routines.

Note 1: Follow the BIOS recovery instructions below for BIOS recovery when the main BIOS block crashes.

Note 2: When the BIOS recovery block crashes, you will need to follow the procedures to make a Returned Merchandise Authorization (RMA) request. (For a RMA request, please see section 3.5 for more information). Also, you may use the Supermicro Update Manager (SUM) Out-of-Band (OOB) (https://www.supermicro.com.tw/products/nfo/SMS_SUM.cfm) to reflash the BIOS.

D.3 Recovering the Main BIOS Block with a USB Device

This feature allows the user to recover the main BIOS image using a USB-attached device without additional utilities used. A USB flash device such as a USB Flash Drive, or a USB CD/DVD ROM/RW device can be used for this purpose. However, a USB Hard Disk drive cannot be used for BIOS recovery at this time.

The file system supported by the recovery block is FAT (including FAT12, FAT16, and FAT32) which is installed on a bootable or non-bootable USB-attached device. However, the BIOS might need several minutes to locate the SUPER.ROM file if the media size becomes too large due to the huge volumes of folders and files stored in the device.

To perform UEFI BIOS recovery using a USB-attached device, follow the instructions below.

1. Using a different machine, copy the "Super.ROM" binary image file into the Root "\" directory of a USB device or a writable CD/DVD.

Note 1: If you cannot locate the "Super.ROM" file in your drive disk, visit our website at www.supermicro.com to download the BIOS package. Extract the BIOS binary image into a USB flash device and rename it "Super.ROM" for the BIOS recovery use.

Note 2: Before recovering the main BIOS image, confirm that the "Super.ROM" binary image file you download is the same version or a close version meant for your motherboard.

1. Insert the USB device that contains the new BIOS image ("Super.ROM") into your USB drive and reset the system when the following screen appears.

SUPERIOR BMC IP:10.132.161.13 PEI--Could Not Find Recovery Image... 87

1. After locating the healthy BIOS binary image, the system will enter the BIOS Recovery menu as shown below.

Optin Setup Utility - Copyright (C) 2017 American Megatrends, Inc. Main Advanced Event Log (DW) Recovery Security Boot Save & Exit Please select blocks you want to update Reset OK Boot Block Update [Enabled] ▶ Proceed with flash update Set this option to reset NWSM to default values +: Select Screen T4: Select Item Enter: Select +/- Change Out. F1: General Help F2: Previous Values F3: Optimized Defaults F4: Save & Exit ESC: Exit Version 2.19.1766. Copyright (C) 2017 American Megatrends, Inc.

Note: At this point, you may decide if you want to start the BIOS recovery. If you decide to proceed with BIOS recovery, follow the procedures below.

1. When the screen as shown above displays, use the arrow keys to select the item "Proceed with flash update" and press the key. You will see the BIOS recovery progress as shown in the screen below.

Note: Do not interrupt the BIOS flashing process until it has completed.

Aotia Setup Utility - Copyright (C) 2017 American Megatrends, Inc. No Security WARNING System firmware is being updated. Keyboard is locked. DO NOT TURN THE POWER OFF III Drive firmware update is completed press any key to reboot the system Program new data Write new boot block... 178 +: Select Screen F4: Select item Enter: Select +/-: Change Opt. F1: General Help F2: Previous Values F3: Optimized Defaults F4: Save a Exit ESC: Exit Version 2.19.1266. Copyright (C) 2017 American Megatrends, Inc.

1. After the BIOS recovery process is complete, press any key to reboot the system.

Roto Setup Utility - Copyright (C) 2017 American Regatrends, Inc. Recovery WARNING System firmware is being updated. Keyboard is locked... DO NOT TURN THE POWER OFF!! Once firmware update is completed press any key to reboot the system Flash update Flash update completed. Press any key to reset the system #: Select Screen F#: Select Item Enter: Select +/-: Change Opt. F1: General Help F2: Previous Values F3: Optimized Defaults F4: Save & Exit ESC: Exit Version 2.15.1266; Copyright (C) 2017 American Regatrends, Inc.

1. Using a different system, extract the BIOS package into a USB flash drive.
2. Press continuously during system boot to enter the BIOS Setup utility. From the top of the tool bar, select Boot to enter the submenu. From the submenu list, select Boot Option #1 as shown below. Then, set Boot Option #1 to [UEFI AP:UEFI: Built-in EFI Shell]. Press to save the settings and exit the BIOS Setup utility.

Auto Setup Utility - Copyright (C) 2017 American Megatrends, Inc. Main Advanced Event Logs IPM Security BOOT Save & Exit Boot Configuration Boot mode select [DUAL] LEGACY to EFI support [Disabled] FIXED BOOT ORDER Priorities Boot Option #1 [DEFI AP:DEFI: No...] Boot Option #2 [OSV/DVD] Boot Option #3 [USB Hard Disk] Boot Option #4 [USB CD/DVD] Boot Option #5 [USB Key:SSND[isk] Boot Option #6 [USB Floppy] Boot Option #7 [USB Lan] Boot Option #8 [Network:IDA GE SI...] Boot Option #9 [UEFI Hard Disk] Boot Option #10 [UEFI CD/DVD] Boot Option #11 [UEFI USB Hard Disk] Boot Option #12 [UEFI USB CD/DVD] Boot Option #13 [UEFI USB Key:UEFI...] Boot Option #14 [UEFI USB Floppy] Boot Option #15 [UEFI USB Lan] Boot Option #16 [DEFI Network] Boot Option #17 [Hard Disk] ▶ Pick New Boot Option Sets the system boot order <<: Select Screen T4: Select Item Enters Select +/-: Change Dot. F1: General) Help F2: Previous Values F3: Optimized Defaults F4: Save & Exit EOG: Exit Version 2.19.1266. Copyright (C) 2017 American Megatrends, Inc.

1. When the UEFI Shell prompt appears, type fs# to change the device directory path. Go to the directory that contains the BIOS package you extracted earlier from Step 6. Enter flash.nsh BIOSname.### at the prompt to start the BIOS update process.

LEFI Interactive Shell v2.1 ESK 11 LEFI v2.50 (American Regatrends, 0x0089XXC) Mapping table: F30: A106(0):H0D(F0):BLK1; PcJW001(0x0)/PcI(0x14,0x0)/US$(0x11,0x0)/H01_MER_0xG7901D2_0xB06_0x1 OR3532) BLK0: B106(0): PcJW001(0x0)/PcI(0x14,0x0)/US$(0x11,0x0) Press ESCR is 1 seconds to skip startup.msh or any other key to continue. Shell (f30) F30: CD BLKOS F30:VAFUOOS> CD SKAPME2_0362017 F30:VAFUOOS SKAPME2_0362017 flash.nsh X13P07.5X

Note: Do not interrupt this process until the BIOS flashing is complete.

Done. [Access Ows Part Ex] Index 0x51: 0x18 Done. ********************************************************************** * Program BIOS and ME (including RDT) regions... ********************************************************************** AMI Firmware Update Utility v5.09.01.1317 Copyright ©2017 American Regatrends Inc. All Rights Reserved. OPUD = 50652 Reading Flash ....... done - ME Data Size checking . ok - PFS checksums ....... ok - Check RowLayout ....... ok, Erasing Boot Block ....... done Updating Boot Block ....... done Verifying Boot Block ....... done Zrading Mesh Block ....... 0x00152000 (00)

1. The screen above indicates that the BIOS update process is complete. When you see the screen above, unplug the AC power cable from the power supply, clear CMOS, and plug the AC power cable in the power supply again to power on the system.

Verifying HCB Book ....... done - Update success for FSK - Update success for EE - - Successful Update Recovery Loader to (PRx1) - Successful Update MFSB11- - Successful Update RTM11- - Successful Update NFS, [081 and 19021] - Successful Update FLOD and UTOX1 - ME entire image update success ll WAVRNDK : System must power-off to have the changes take effect! Moving F501:AFUD005/SJPM2E_03162037/40x64,efi -> F501:AFUD005/SJPM2E_03162017-y df.sac - [c4] Moving F501:AFUD005/SJPM2E_03162037/aufemfix4,efi -> F501:AFUD005/SJPM2E_0316201 ?afuefi.sac - [c4] ********************************************************************** * * Please ignore this 'Shell cannot read from file - device error' * Warning message due to it does not impact flashing process. * ********************************************************************** initating * Delete successful. F501->

1. Press continuously to enter the BIOS Setup utility.
2. Press to load the default settings.
3. After loading the default settings, press to save the settings and exit the BIOS Setup utility.

Appendix E

IPMI Crash Dump

In the event of a processor internal error (IERR) that crashes your system, you may want to provide information to support staff. You can download a crash dump of status information using IPMI. The IPMI manual is available at https://www.supermicro.com/solutions/IPMI.cfm.

Check IPMI Error Log

1. Access the IPMI web interface.
2. Click the Server Health tab, then Event Log to verify an IERR error.

Host Identification Server: 172 831 040 125 User: ADMIN (Administrator) System Server Health Configuration Remote Control Virtual Media Maintenance Miscellaneous Help Server Health Sensor Readings Event Log Event Log For more special event log settings, please click here. This page displays events from the system's event log. You can choose a category from the pull-down box to filter the events and also sort them by clicking on a column header. Clear Event Log Save Select an event log category: All Events Event ID Time Stamp Sensor Name Sensor Type Description 1 2017/10/19 15:38:37 Processor IERR - Assertion 2 2017/10/19 15:50:20 Processor IERR - Assertion

Figure E-1. IPMI Event Log

In the event of an IERR, the BMC executes a crash dump. You must download the crash dump and save it.

Downloading the Crash Dump File

1. In the IPMI interface, click the Miscellaneous tab, then the Trouble Shooting option.
2. Click the Dump button and wait five minutes for the file to be created. (No confirmation message will appear.)
3. Click the Download button and a Save As dialog appears.
4. Save the zipped dump file, noting the name and location.

SUPERMICRO Host Identification Server: 172.031.040.125 User: ADMIN (Administrator) System Server Health Configuration Remote Control Virtual Media Maintenance Miscellaneous Help Miscellaneous Activate License Post Snooping SMC RAKP Trouble Shooting UID Control Trouble Shooting This page provides trouble shooting file download System auto reset Dump Download

Figure E-2. IPMI Crash Dump Download

Note: The System auto reset check box dictates behavior after an IERR. If checked, the system will restart automatically, and the dump file will be erased. If not, the system remains in a failed state. Do not check this box until after the dump file has been sent to Support.

Appendix F

CPU-Based RAID for NVMe

Intel® Virtual RAID on CPU (Intel VROC) is an enterprise RAID solution for NVMe SSDs directly attached to Intel Xeon Scalable processors. Intel Volume Management Device (VMD) is an integrated controller inside the CPU PCI-E root complex.

• A single processor supports up to 12 NVMe SSDs and up to 6 RAID arrays.
• A dual processor system supports up to 24 NVMe SSDs and 12 RAID arrays.

Strip sizes are 4K, 8K, 16K, 32K, 64K, 128K.

Requirements and Restrictions

• Intel VROC is only available when the system is configured for UEFI boot mode.
• To enable the mdadm command and support for RSTe, install the patch from

- Linux: https://downloadcenter.intel.com/download/28158/Intel-Virtual-RAID-on-CPU-Intel-VROC-and-Intel-Rapid-Storage-Technology-enterprise-Intel-RSTe-Driver-for-Linux-

- Windows: https://downloadcenter.intel.com/download/28108/Intel-Virtual-RAID-on-CPU-Intel-VROC-and-Intel-Rapid-Storage-Technology-enterprise-Intel-RSTe-Driver-for-Windows-

• To enable Intel VROC, a hardware key must be inserted on the motherboard, and the appropriate processor's Virtual Management Devices must be enabled in the BIOS setup.
• It is possible to enable Intel VROC without a hardware key installed, but only RAID0 will be enabled.
• Intel VROC is not compatible with secure boot. This feature must be disabled.
• When creating bootable OS RAID1 devices, you must have both devices on the same CPU, and a VMD on that CPU.
• Spanning drives when creating RAID devices is not recommended to due to performance issues, even though it is supported.

Supported SSDs and Operating Systems

To see the latest support information: https://www.intel.com/content/www/us/en/support/articles/000030310/memory-and-storage/ssd-software.html

Additional Information

Additional information is available on the product page for the Supermicro add-on card and the linked manuals.

www.supermicro.com/products/accessories/addon/AOC-VROCxxxMOD.cfm

F.1 Hardware Key

The Intel VROC hardware key is a license key that detects the Intel VROC SKU and activates the function accordingly. The key must be plugged into the Supermicro motherboard (connector JRK1). The key options are:

Intel® VROC Keys
VROC Package Description Part Number Intel MM Number
Standard	RAID 0, 1, 10Supports 3rd party SSDs	AOC-VROCSTNMOD 95	1605
Premium	RAID 0, 1, 5, 10Supports 3rd party SSDs	AOC-VROCPREMOD 95	1606
Intel SSD only	RAID 0, 1, 5, 10Supports Intel SSDs only	AOC-VROCINTMOD 95	822

VROC_HW_KEY PIN 1 GND KEY1 GND KEY2

Figure F-1. Intel® VROC RAID Key and Motherboard Connector JRK1

F.2 Enabling NVMe RAID

RAID for NVMe SSDs must be enabled through the UEFI BIOS.

1. Install the patch as described in the Restrictions and Requirements section on a previous page.
2. Reboot the server.
3. Press [DEL] key to enter BIOS.
4. Switch to Advanced > Chipset Configuration > North Bridge > IIO Configuration > Intel® VMD Technology > CPU1.
5. Enable the VMD according to the following rules.

- For U.2 NVMe, enable all the sub-items under each PStack.

- For M.2 NVMe or NVMe AIC, enable the VMD according to which AOC card/slot it used. An example U.2 configuration follows.

VHD Config for P3stack Install VHD for Volume Management Device (Disable) VHD Config for P3stack1 Install VHD for Volume Management Device (Disable) RDC-100M-6 NAME: VHD (Disable) NOT NULL CAPSULE (Disable) VHD Config for P3stack2 Install VHD for Volume Management Device (Disable) Enable/Disable SelectV Volume Management Device Technology in this Stack. << Select Screen File Select Data Delete Click +/- Change Out. Fid General mail Fid Previous Values For Unloaded Defaults Fid Save & Exit EIC: Exit Version 3.19.2008 Copyright HCI 510A American Reporters, Inc.

Figure F-2. A BIOS VMD Setting Example (different server)

1. Press [F4] to save the configuration and reboot the system.
2. Press [DEL] to enter BIOS.
3. Switch to Advanced > Intel(R) Virtual RAID on CPU > All Intel VMD Controllers > Create RAID Volume.
4. Set Name.
5. Set RAID Level.
6. If cross-controller RAID is required, select Enable RAID spanned over VMD Controller as shown in Figure F-4.

AutoCAD Utility - Copyright (C) 2018 American Markets, Inc. Advanced Create RAID Volume Name: AVGO (eval) Volume: AVGO (visual) Enable RAID spread over IWD Controller [ ] Select Blocks: AVGO SUPERCINHAST QUALITY/ISUROPAPIE [ ] AVGO SUPERCINHAST QUALITY/ISUROPAPIE [ ] AVGO SUPERCINHAST QUALITY/ISUROPAPIE [ ] Strip Size: Capacity: 100% [100%] Capacity: 100% [100%] ▶ Create Volume K - to Select Data →: Select Screen F4: Select 2 items Enter: Select +/-: Change Rate F1: General Rate F2: Previous Values F3: Optimal Defaults F4: Save & Exit F5: Exit

Figure F-3. Created Volume without enabling RAID spanned over VMD controller

Auto Setup Utility - Copyright (C) 2008 American MegaTrends, Inc. Advanced Create R&D Volume Name: M&D Levels: Enable R&D spread over MR Controller: DC Select Summary DATE: SUPEREXCHARTS CAPRIFICATION DATE: SUPEREXCHARTS CAPRIFICATION DATE: SUPEREXCHARTS SEPARATION DATE: SUPEREXCHARTS SEPARATION Strip Size: Capacity: 14510.00 * Create volume X = to Select Data -> Select Screen F1: Select Item Enter: Select +/- Change Set... F2: General Help F3: Previous Values F4: Default Defaults F5: Save A Exit G3: Exit Version 2.13.DSA, Copyright © 2008 American MegaTrends, Inc.

Figure F-4. Created Volume with enabling RAID spanned over VMD controller

1. Select specific disks for RAID with an [X].

• RAID0: Select at least two [2 - 24] disks
• RAID1: Select only two disks
• RAID5: Select at least three [3 - 24] disks
• RAID10: Select only four disks

1. Select Strip Size (Default 64KB).
2. Select Create Volume.
3. If another RAID is needed, start again at step 6.
4. Press [F4] to save and reboot.

F.3 Status Indications

An LED indicator on the drive carrier shows the RAID status of the drive.

Drive Carrier Status LED Indicator
Status State (red)
Normal function Off
Locating 4 Hz blink
Fault Solid on
Rebuilding 1 Hz Blink

IBPI SFF 8489 Defined Status LED States

F.4 Hot Swap Drives

Intel VMD enables hot-plug and hot-unplug for NVMe SSDs, whether from Intel or other manufacturers. Under vSphere ESXi, several steps are necessary to avoid potential stability issues. See the information at link [1] below.

Hot-unplug

1. Prevent devices from being re-detected during rescan:

esxcli storage core claiming autoclaim --enabled=false

1. Unmount the VMFS volumes on the device. Check [2] for details.
2. Detach the device. Check [3] for details.
3. Physically remove the device.

Hot-plug

• Physically install the device.

ESXi will automatically discover NVMe SSDs, but a manual scan may be required in some cases.

Related Information Links

[1] https://kb.vmware.com/s/article/2151404
[2] https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.storage.doc/GUID-1B56EF97-F60E-4F21-82A7-8F2A7294604D.html
[3] https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.storage.doc/GUID-F2E75F67-740B-4406-9F0C-A2D99A698F2A.html