OpenSSH 9.0 includes a change that is incompatible with IBM Spectrum Scale. Ensure that OpenSSH 9.0 is not used with IBM Spectrum Scale. Earlier OpenSSH packages from Cygwin work. It is expected that this issue will be resolved with OpenSSH 9.1.

These coexistence rules also apply for remote cluster access (multi-cluster remote mount). A node running release N-2 cannot perform a remote mount from a cluster which has nodes running release N, and vice versa.

• Exporting using NFS V4 is supported starting with IBM Spectrum Scale V4.1 or later.
• CNFS over IPV6 is only supported with IBM Spectrum Scale V4.1 or later.
• It is very important to make all the nodes in the same group as identical as possible - from the hardware and software running on them to the configuration of IBM Spectrum Scale, NFS, and the network.
• CNFS is not supported on the Ubuntu distribution.
• CNFS is not supported on SLES 15 and later versions.
• CNFS is not able to export a remotely mounted filesystem.
• Clustered NFS
• Integrated protocols (CES and NFS), which support NFSv3, NFSv4.0 and NFSv4.1.
  Note: Some features of NFSv4.1 are not supported. For example, pNFS.
  Enhancements to the support of Network File System (NFS) V4 are available on:
  • AIX V6.1 or AIX V7.1.
  • The following Linux distributions:
    • RHEL 5.5 and later, 6. x , and 7 x .
    • SLES 11 SP1 and later, and SLES 12
    Restrictions include:
    • To support NFsv4 ACLs, the package nfs4-acl-tools must be installed.
    • Windows-based NFSv4 clients are not supported with Linux/NFSv4 servers because of their use of share modes.
    • If a file system is to be exported through CNFS (the Linux kernel NFSserver), then it must be configured to support POSIX ACLs (with -k all or -k posix option). This is because NFSv4/Linux servers will only handle ACLs properly if they are stored in GPFS as posix ACLs. On the other hand, if a file system is to be exported through CES (Using Samba and NFS Ganesha), then it must be configured to only support NFSv4 ACLs (with -k nfs4 ). This is because the CES stack has only been qualified with NFSv4 ACLs.
    • Starting with Linux kernel version 2.6, an fsid value must be specified for each GPFS file system that is exported on NFS. For example, the format of the entry in /etc/exports for the GPFS directory /gpfs/dir1 might look like this:

      /gpfs/dir1 cluster1(rw,fsid=745)

      For further details see the Linux export considerations at http://www-01.ibm.com/support/knowledgecenter/STXKQY_4.1.1/com.ibm.spectrum.scale.v4r11.adm.doc/bl1adm_nfslin.htm
    • Concurrent AIX/NFSv4 servers, Samba servers and GPFS Windows nodes in the cluster are allowed. NFSv4 ACLs may be stored in GPFS filesystems via Samba exports, NFSv4/AIX servers, GPFS Windows nodes, ACL commands of Linux NFSv3 and ACL commands of GPFS. However, clients of Linux v4 servers will not be able to see these ACLs, just the permission from the mode.
    For more information on the support of NFS V4, please see the Spectrum Scale documentation updates file at
    • For IBM Spectrum Scale V4.1.1 and later, at http://www.ibm.com/support/knowledgecenter/STXKQY/ibmspectrumscale_welcome.html
    • For GPFS V3.5 and V4.1, at http://www-01.ibm.com/support/knowledgecenter/SSFKCN/gpfs_welcome.html
    Q2.11:
    Are there any considerations for the use of the Persistent Reserve support in IBM Spectrum Scale?
    A2.11:
    Considerations for the use of Persistent Reserve include:

    Support for Persistent Reserve requires:

    For V3.5 support on AIX V6.1 requires APAR IZ57224

    AIX 6.1 TL7 + Service Pack 4 is required to support Persistent Reserve without SDDPCM.

    For V3.5 , V4.1 or V4.2 support on AIX V7.1, refer to the storage documentation to install the correct multipath driver.

    The use of Persistent Reserve is supported on GPFS tie-breaker disks with GPFS V3.5.0.21, or later, and IBM Spectrum Scale V4.1.0.4, or later.

    Note: See the question What are the current requirements/limitations for using the Cluster Configuration Repository (CCR) ?

    For the Activate Persist Through Power Loss (APTPL) feature:

    On Linux, if the storage is capable of supporting APTPL, GPFS V3.5.0.15, or later supports this feature.

    Starting with 3.5.0.16, it is possible to have a descOnly disk that resides on a device that does not support SCSI-3 Persistent Reserve while allowing Persistent Reserve to be used on other disks in the same file system. The lack of Persistent Reserve support for the descOnly disk will not result in fast failover being disabled.

    Also see the question What devices does GPFS support with SCSI-3 Persistent Reservations?

    Q2.12:
    What are the requirements/limitations for using native encryption in IBM Spectrum Scale Advanced Edition or Data Management Edition?
    A2.12:
    Considerations for the use of native encryption (encryption of data at rest on GPFS disks) in IBM Spectrum Scale Advanced Edition include:
    • The installation and use of either IBM Security Key Lifecycle Manager (ISKLM) V2.6 or later, or Vormetric Data Security Manager (DSM) V6.2 or later, is required for each node that acts as a key server. Key server nodes are not required to be members of the IBM Spectrum Scale cluster(s) that use them.
      For ISKLM:

      • IBM Security Guardium Key Lifecycle Manager (GKLM) V4.2 is not supported with IBM Spectrum Scale Encryption feature.
      • IBM Security Guardium Key Lifecycle Manager (GKLM) V4.1.1 is supported with the IBM Spectrum Scale Encryption feature.
      • IBM Security Guardium Key Lifecycle Manager (GKLM) V4.1.0.1 (IF01) is supported with the IBM Spectrum Scale Encryption feature.
      • ISKLM is not shipped with nor licensed with IBM Spectrum Scale and must be purchased separately.
      • ISKLM V2.6, or later of the server software (D0887LL) must be installed on each node that acts as a key server.
      • See the IBM Security Key Lifecycle Manager documentation for details about licensing that offering.
      For Vormetric:
      • Vormetric Data Security Manager is not shipped with nor licensed with IBM Spectrum Scale and must be purchased separately. Contact Vormetric directly to purchase.
      • Vormetric Data Security Manager is not supported on nodes installed with Linux on Z.
      • Every node that accesses the encrypted data must be running the Advanced Edition or Data Management Edition of IBM Spectrum Scale.
      • Every node that accesses the encrypted data, and also nodes which play a management role in the file system (such as manager node, an NSD server, or a node which participates in the restripe of a file system), must have network connectivity to the key server.
      • IBM Spectrum Scale Client nodes do not require a key server license.
      • Vormetric DSM V6.0.2, V6.0.3, and V6.1.x releases are not supported with IBM Spectrum Scale encryption. The user interface in these releases does not support the creation of KMIP objects such as the Master Encryption Keys (MEKs) that are used by IBM Spectrum Scale encryption. For more information, see https://www-01.ibm.com/support/docview.wss?uid=ibm10734479 .
      • Vormetric DSM V6.2 user interface supports the creation of KMIP objects such as the Master Encryption Keys (MEKs) used by IBM Spectrum Scale encryption. IBM Spectrum Scale encryption is supported with Vormetric DSM V6.2.
      Current limitations of the IBM Spectrum Scale encryption function include:
      • Only user data is encrypted. The encryption of directories or other metadata is not supported.
      • Extended attributes are not encrypted.
      • Data, which is backed up, is in cleartext unless encryption is supported by the backup system.
      • Data, which is migrated to tape using software such as IBM Spectrum Protect or IBM Spectrum Archive, is in cleartext unless the tape system and the connection between them (if Ethernet or InfiniBand) provide encryption.
      • Encryption is not supported on Windows. The encryption function should be disabled when Windows nodes are in the cluster.
      • FIPS mode is supported on the POWER8® and later processors in little endian mode in IBM Spectrum Scale V4.2.1 and later.
      • The contents of encrypted files are placed into a local read-only cache (LROC) based on the settings of the lrocEnableStoringClearText configuration option. For more information, see the "Encryption and local read-only cache (LROC)" section.
      • For more information, see Encryption requirements and limitations and Q6.12 How should IBM Spectrum Scale Advanced Edition or Data Management Edition be configured to only use FIPS 140-2-certified cryptographic engines?
        Q2.13:
        Are there any considerations when utilizing the Simple Network Management Protocol (SNMP)-based monitoring capability in IBM Spectrum Scale?
        A2.13:
        Considerations for the use of the SNMP-based monitoring capability include:

        The SNMP collector node must be a Linux node in your GPFS cluster. GPFS utilizes Net-SNMP which GPFS does not support on AIX.

        Support for ppc64 requires the use of Net-SNMP 5.4.1. Binaries for Net-SNMP 5.4.1 on ppc64 are not available. You will need to download the source and build the binary. Go to http://net-snmp.sourceforge.net/download.html

        If the monitored cluster is relatively large, you need to increase the communication time-out between the SNMP master agent and the GPFS SNMP subagent. In this context, a cluster is considered to be large if the number of nodes is greater than 25, or the number of file systems is greater than 15, or the total number of disks in all file systems is greater than 50. For more information see Configuring Net-SNMP in the IBM Spectrum Scale: Problem Determination Guide .

        SNMP-based monitoring has not been tested in clusters composed of more than 127 nodes.
        Current limitations and advisories include:

        • Beginning with IBM Spectrum Scale V4.2, in file systems that are managed by an HSM system, mmbackup will skip over candidates for backup that are migrated offline to avoid causing a recall storm. Instead, records for these files will be added to a file in the root of the fileset called mmbackup.hsmMigFiles. name of server . System managers should recall these changed files online to allow mmbackup to properly protect them in the next invocation.
        • File systems with IBM Spectrum Protect for Space Management that have unlinked filesets, will be required to link all filesets when issuing the mmbackup command for the first time after you upgrade GPFS cluster from GPFS 3.5.0.10 or lower, to GPFS 3.5.0.11 or higher. If you have any concerns regarding this requirement, please contact GPFS service.
        • In the United States contact us toll free at 1-800-IBM-SERV (1-800-426-7378)
        • In other countries, contact your local IBM Service Center
        • Use of the IBM Spectrum Protect Backup-Archive client option SKIPACLUPDATECHECK with the mmbackup command requires IBM Tivoli Storage Manager release 6.4.1.0 or later.
          Note: Beginning with Version 7.1.3, IBM Tivoli Storage Manager is now IBM Spectrum Protect.
        • The GPFS mmbackup command is not integrated with the IBM Spectrum Protect for Space Management-Multi HSM Server feature. See Managing a file system with multiple Tivoli Storage Manager servers .
        • Restoring a file via a node that has a different architecture than the one used to do the backup could cause the associated ACL to be corrupted.

          For example, if a file was backed up using an x86_64 node, and then restored using a ppc64 node, this could cause its ACL to be corrupted (caused by differences in endianness of the architectures which is not supported by the GPFS APIs used in the restore operation). It is recommended that backup and restore operations be done on similar types of nodes.

        • The mmbackup command supports backup of a whole file system from a global snapshot.

          The mmbackup -S snapshot command option is supported with IBM Spectrum Scale V4.1.1 on either a global snapshot for the whole file system or for a fileset backup if the fileset was captured in that snapshot. It is also supported with a fileset snapshot for a fileset backup, providing the name of the snapshot is unique among all snapshot names. Do not use the same snapshot name for multiple snapshots.

          The mmbackup -S snapshot command option is supported with GPFS V3.5.0.3 or later. GPFS V3.4 and GPFS V3.5.0.2 or lower do not support backup from a snapshot. If the snapshot directory for global snapshots and the directory for fileset level snapshots are different, then GPFS V3.5.0.4 or higher level is required.
        • Doing backup from a snapshot in an IBM Spectrum Protect for Space Management managed file system could cause recall of migrated files.

          In an HSM managed file system such as IBM Spectrum Protect for Space Management, using mmbackup to back up from a snapshot could cause the recall of migrated files if the migration was done after the snapshot was taken. This is due to the fact that a snapshot is a static view of the file system which does not reflect migration state changes. To avoid recalling data from migrated files, create the snapshot and complete the backup operation before migrating files or make sure that migration is done before the snapshot is taken for a backup operation. Backup operations will not recall files if the snapshot captured the files in their migrated state. Until and unless the migrated file stubs are removed from the live file system. In this case a recall will be required to populate the contents of the snapshot view of the files. If a snapshot exists, consider recalling files by using the IBM Spectrum Protect for Space Management "tape optimized recall" function before deleting migrated files from the active file system.

        • Doing backup from a snapshot in an IBM Spectrum Protect for Space Management managed file system could cause failure to backup migrated files.
        • The use of unsupported characters in the names of files or directories will cause failures.

          mmbackup uses the IBM Spectrum Protect Backup Archive client to backup data to the IBM Spectrum Protect server. As IBM Spectrum Protect currently does not support all special characters in file or directory names, they cannot be supported by mmbackup . If special characters are used in the names of files or directories backed up by the mmbackup command, failures will result. Known special characters which can cause problems include: * , ? , " , ' , control-X, control-Y , carriage return and the new line character. Use of the IBM Spectrum Protect options QUOTESARELITERAL and WILDCARDSARELITERAL along with the --noquote command line option to mmbackup will allow support for all special characters except carriage return, new line, control-X, and control-Y.

        • Beginning with IBM Spectrum Scale V4.1.1, backup of either the entire file system or a selected fileset is supported. Nested fileset arrangements where one fileset is linked inside another are not supported by mmbackup on a fileset. Nesting remains supported for whole file system mmbackup . The first mmbackup of any fileset must be made using the option -t full to avoid causing accidental invalidation of existing backups that may exist of previously existing nested filesets.
        • Differences in the way the mmbackup command and IBM Spectrum Protect process the include and exclude statements in the dsm.sys configuration file may cause files or directories to be included or excluded unexpectedly.
          Known differences in processing include, but are not limited to:
        • The mmbackup command does not support exclude.archive , exclude.file.spacemgmt , exclude.spacemgmt , exclude.fs .
        • Whether or not there is a / at the end of an exclude.dir affects the way mmbackup decides what files or directories are excluded.
        • exclude.file may cause incorrect files to be backed up if the pattern presents a wildcard at the end.
        What are the current limitations and advisories for using Scale Out Backup and Restore (SOBAR) ?
        A2.15:
        Current limitations and advisories include:
        • IBM Spectrum Scale Image Backup and Restore (SOBAR) has been tested in a standalone manner; but must be tested with Data Management/HSM products before deployment by customers with such products in production environments. Customers who are interested in making use of this function should contact scale@us.ibm.com .
        • SOBAR does not support the backup or restore of a file system with Active File Management (AFM) filesets.
        • SOBAR supports backup from a global snapshot. Independent fileset snapshots are not supported.
        Current limitations include:
        • The File Placement Optimizer (FPO) function is supported on IBM Spectrum Scale GPFS V5, V4, and V3.5 for both the Linux and AIX operating systems. For V3.5, the Linux (x86 and Power) operating system requires APAR IV28687 and the AIX operating system requires APAR IV40108.
        • AFM ADR (primary/secondary filesets) is not supported on an FPO enabled file system.
        • With the AFM function, if you want to maintain data locality on both home and cache, they must have the same Failure Group configuration. Additionally, block placement policy must be set via write-affinity-failure-group at both sites.
        • Twin-tailed disks are supported in an FPO pool only when a single NSD server is defined for each disk.
        • The FPO function is not supported on the Debian distribution or Linux on IBM Z.
        • Nodes running the GPFS File Placement Optimizer feature cannot coexist or interoperate with nodes running GPFS V3.4 or earlier releases of GPFS.
        • Contact scale@us.ibm.com if you plan to deploy a cluster with more than 32 nodes in a Shared Nothing Cluster, or SNC, in which no disks in the cluster are served by more than a single node. This includes FPO nodes. Shared Nothing Clusters that are larger than 32 nodes must be reviewed and approved by IBM before deployment. This limitation applies to clusters that have more than 32 nodes that have disks serving a file system with data replication enabled, and these disks are only accessible from a single node.
          Note: You can determine if a given file system has data replication enabled by checking if the -R setting (the maximum number of data replicas) reported by the /usr/lpp/mmfs/bin/mmlsfs command is greater than 1.
        • If a storage pool is FPO-enabled ( allowWriteAffinity=yes ), then layoutMap=cluster must also be specified.
        • With GPFS V3.5, use of the mmrestripefile , mmadddisk -r and the mmrestripefs commands will break the original FPO file's placement.
        • With GPFS V4.1, use of the mmrestripefile -b , mmadddisk -r and the mmrestripefs -b commands will break the original FPO file's placement.
        • With GPFS V4.1, use of the mmrestripefile -r and the mmrestripefs -r commands is supported with locality awareness. Use of the commands with clones and snapshots will break the original FPO file's placement.
        • On clusters with the FPO function enabled, in order to utilize the mmrestorefs command, you must specify the write-affinity-failure-group policy.
        • If the size of a file is less than the value of the block size divided by 32, the write affinity depth policy and the write affinity failure group policy will not be followed. Data is widely striped instead.
        • The setXattr function cannot set the FPO extended attributes writeAffinityDepth , write-affinity-failure-group , and BlockGroupFactor for a clone file or the policy MIGRATE rule. Respectively, setWAD , setWADFG , and setBGF should be used.
        • The extended attributes writeAffinityDepth , write-affinity-failure-group , and BlockGroupFactor are for use only on an FPO pool.
        • Starting in IBM Spectrum Scale 5.0.5, FPO and SNC remain available. However, it is recommended to limit the size of deployments to 32 nodes. There are no plans for significant new functionality in FPO nor increases in scalability. The strategic direction for storage using internal drives and storage rich servers is IBM Spectrum Scale Erasure Code Edition.
        • The FPO configuration is not supported on IBM Spectrum Scale Erasure Code Edition.
          • Preparing for the IBM Spectrum Scale Erasure Code Edition environment: https://www.ibm.com/support/knowledgecenter/STXKQY_BDA_SHR/bl1bda_prepece.htm
          • Restrictions: https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.5/com.ibm.spectrum.scale.v5r05.doc/bl1adv_fporestrictions.htm
          Q2.17:
          What are the current limitations for using the Active File Management (AFM) Async DR function?
          A2.17:
          Limitations are added and deleted from time to time. For more information about the limitations that affect a particular release, see the AFM limitations section under Product Overview > Active File Management in the Knowledge Center or in the IBM Spectrum Scale : Concepts, Planning, and Installation Guide .
          A2.18:
          Limitations are added and deleted from time to time. For more information about the limitations that affect a particular release, see the AFM limitations section under Product Overview > Active File Management in the Knowledge Center or in the IBM Spectrum Scale : Concepts, Planning, and Installation Guide .
          Q2.19:
          What are the current limitations common to both the Active File Management (AFM) and AFM DR functions?
          A2.19:
          Limitations are added and deleted from time to time. For more information about the limitations that affect a particular release, see the AFM and AFM DR limitations section under Product Overview > Active File Management in the Knowledge Center or in the IBM Spectrum Scale : Concepts, Planning, and Installation Guide .
          A2.20:
          For the current recommendations regarding the transport protocol for AFM and AFM DR data transfers, see The backend protocol - NFS versus NSD in the IBM Spectrum Scale: Concepts, Planning, and Installation Guide .
          A2.21:
          For mixed clusters containing GPFS V3.5, IBM Spectrum Scale 4.1.0, 4.1.0.4, 4.1.1, or V4.2 nodes, it is recommended that for both fileset and global snapshot restore, the mmrestorefs command is issued from a node running the most recent version of Spectrum Scale. Version 4.1.1 and later, provides some new snapshot restore functionality so IBM Spectrum Scale attempts to intelligently use the latest features.
          Note:
          1. IBM Spectrum Scale V4.2 only interoperates with IBM Spectrum Scale V4.1
          2. The mmrestorefs command is not supported on the Express Edition.
          3. To improve performance support for the –N parameter on the mmrestorefs command has been phased in over the past few releases.
          4. In GPFS 3.5 and earlier there is no –N parameter
          5. In GPFS 4.1 –N can be used for fileset snapshot restore only
          6. In IBM Spectrum Scale 4.1.1 and later –N can be used for both fileset and global snapshot restores
          7. The file system must be mounted.
          8. By default the restore is performed on all nodes running the latest level of code.
          9. The file system must be mounted.
          10. By default the restore is performed on all nodes running the latest level of code.
          11. A minimum of IBM Spectrum Scale V4.1.0.1.
          12. Local read-only cache is only supported on Linux x86 and Power.
          13. The minimum size of a local read-only cache device is 4 GB.
          14. The local read-only cache requires memory equal to 1% of the local read-only device's capacity.
          Note: Use of local read-only cache does not require a server license
          What are the current requirements/limitations for using the Cluster Configuration Repository (CCR) ?
          A2.23:
          The current requirements/limitations for using the Cluster Configuration Repository (CCR) include:
          IBM Spectrum Scale V4.1.0: The Disaster Recovery procedures described in the Advanced Administration Guide are not supported in a cluster with CCR enabled:
          • Do not run mmchcluster --ccr-enable for existing clusters
          • Use mmcrcluster --ccr-disable for new clusters
          The current requirements/limitations for using Ubuntu include:
          Secure Boot needs to be disabled on Ubuntu.
          The minimum level of Ubuntu supported is 14.04.1.
          Only IBM Spectrum Scale 4.1.0.8 or later, is supported with 14.04.2.
          Only IBM Spectrum Scale 4.1.1.9/4.2.1.1 or later is supported with 14.04.4/16.04.
          The minimum kernel level supported is 3.13.0.32.
          Only IBM Spectrum Scale 4.2.3.10/5.0.1.2 or later is supported with 18.04.1.
          Only IBM Spectrum Scale for Linux on Z 4.2.3.10/5.0.1.2 or later is supported with 18.04.
          P8 is supported with Little Endian only and only with GPFS for Linux on System p base RPMs dated January 2015 (GPFS V4.1.0.5 or later).
          If you have Software Maintenance Agreement (SWMA) for your products ordered through AAS/eConfig or IBM Subscription and Support (S&S) for orders placed through Passport Advantage, you may log into the respective systems and upgrade your level of GPFS:
          For products ordered through AAS/eConfig, please log into the Entitled Software page at: https://www-05.ibm.com/servers/eserver/ess/OpenServlet.wss
          For products ordered through Passport Advantage, please log into the site at: http://www.ibm.com/software/lotus/passportadvantage/
          GPFS V3.5.0.22 or later is only supported on x86_64 architecture
          When issuing make World for Ubuntu 14.04.1, this warning will appear but can be disregarded because kdump-kern-dummy.ko is not utilized by GPFS.
          WARNING: ".TOC." [/usr/lpp/mmfs/src/gpl-linux/kdump-kern-dummy.ko] undefined!
          As Tivoli Storage Manager (TSM) does not support Ubuntu, GPFS commands that utilize TSM are not supported on Ubuntu.
          Note: Beginning with Version 7.1.3, IBM Tivoli Storage Manager is now IBM Spectrum Protect.
          If you use CNFS or CES features under Ubuntu, verify that the iputils-arping package is installed. See the Software requirements page for more information.
          Q2.25:
          What are the current requirements/limitations for IBM Spectrum Scale for Linux on Z?
          A2.25:
          The current requirements and limitations for IBM Spectrum Scale for Linux on Z include:
          • A leapp upgrade from Red Hat Enterprise 7.x to 8.x is not supported.
          • File Placement Optimizer is not supported.
          • For support of backup and restore functions with IBM Spectrum Scale for Linux on Z, see the following support matrices:
            Note: Starting with IBM Spectrum Scale V5.0.0, the IBM Spectrum Protect for Space Management client is no longer supported on IBM Spectrum Scale for Linux on Z.
          • For the IBM Spectrum Protect Backup Archive client, see Hardware and software requirements for IBM Spectrum Protect™ Linux zSeries Backup-Archive and API Client .
          • For the IBM Spectrum Protect for Space Management client, see IBM Spectrum Protect™ for Space Management (HSM) requirements for Linux on IBM z Systems® .
          • For supported storage, see the What disk hardware has IBM Spectrum Scale been tested with? and the Does IBM Spectrum Scale for Linux on Z support Direct Attached Storage Devices (DASD)? questions.
          • Support for stretched cluster with synchronous mirroring utilizing block-level replication:
            • For IBM Spectrum Scale 5.0.0 and later, with distances up to 300 km.
            Note:
            Central Processor Assist for Cryptographic Function (CPACF) is supported. CPACF is IBM Z hardware encryption acceleration. It is incorporated in the central processors that are shipped with IBM Z. To benefit from the CPACF, you must install LIC internal feature 3863 (Crypto Enablement feature), which is available free of charge. By default, IBM Z is delivered to customers without this feature unless it is ordered explicitly by the customer. The installation of this feature at a future time is nondisruptive.
            IBM z15 and z16 offer the Integrated Accelerator for zEnterprise®® Data Compression (zEDC). This feature is enabled by default starting from IBM Spectrum Scale 5.1.0 when the CPU feature 'dflt' is listed in /proc/cpuinfo . Compression options such as z, zfast, alphae and alphah benefit from this feature. The feature also depends on your installed zlib version. For more information, see https://linux.mainframe.blog/zlib-acceleration/ for details.
            A2.26:
            The Highly-Available Write Cache (HAWC) function available with IBM Spectrum Scale V4.1.1.1 or later, reduces the latency of bursts of small write requests by buffering them in fast storage such as SSDs.
            Note: If you plan on using the HAWC function on client nodes, V4.1.1.2 is required.
            The HAWC function can benefit numerous applications such as VMs, appending to logs and many more. If a file system's metadata is already stored on fast storage such as SSDs, then the feature can be simply enabled with very little effort. If not, then a new 'fast storage' pool must be created on either one or more NSD servers or on the clients themselves. HAWC is controlled via the file system parameter write-cache-threshold and can also be used with existing as well as new file systems. For more information, see the IBM Spectrum Scale Advanced Administration Guide at http://www-01.ibm.com/support/knowledgecenter/STXKQY/411/com.ibm.spectrum.scale.v4r11.adv.doc/bl1adv_hawc.htm?lang=en
            Q2.27:
            What are the current requirements/limitations for the deadlock amelioration function in IBM Spectrum Scale?
            A2.27:
            The current requirements/limitations for use of the deadlock amelioration function include:
            Deadlock amelioration functions are fully supported in IBM Spectrum Scale V4.
            In a cluster with minReleaseLevel below 4.1.0, that consists of all GPFS 4.1 nodes or a mixture of 4.1 and 3.5 nodes, the deadlock amelioration functions may still work partially. In order to avoid a problem of tracing not being turned off after GPFS code turns it on make sure to have 3.5.0.24 or later, or 4.1.0.7 or later, or have APAR IV69797 applied to all nodes. Running with tracing on could have performance implications.
            Considerations for using the IBM Spectrum Scale GUI include:
            • The GUI is available with the Standard and Advanced Editions for Linux on x86, Linux on Z, and Power (Big Endian and Little Endian).
            • The GUI is supported on RHEL 7.1 or later, SLES 12 SP1 and SP2, and Ubuntu 16.4 on Linux x86, Linux on POWER, and Linux on z platforms. For more information, see 2.1 What is supported on IBM Spectrum Scale for AIX, Linux, Power, and Windows? .
            • The maximum number of nodes supported is:
            • With V4.2.1 and later, 1000 nodes
            • With V4.2, 128 nodes
            • When planning to add GUI nodes with the Installation Toolkit, add them via spectrumscale install or spectrumscale deploy , either before performing an upgrade to 4.2.0.1 or later. Attempting to add GUI nodes during the upgrade itself may result in a failure during the Upgrading Performance Monitoring step.
            • The GUI works with either a client or server license.
            • The GUI depends on a pre-installed PostgreSQL server usually installed already with the operating system installation but if this is not the case if you install the operating system from scratch you need to do this before you install the GUI or the installation fails.
            • Compressing clones and cloning compressed files.
            • Small file compression (files consuming less than 2 sub-blocks, compressing small files into inode).
            • Compression of non-regular files, such as directories.
            • Compression of files in Windows hyper allocation mode.
            • File compression does not compress a memory-mapped file.
            • File compression does not compress a file that is opened for Direct I/O.
            Additionally:
            • Compression is supported in an FPO environment or horizontal storage pools with IBM Spectrum Scale V4.2.1 and later.
            • Compression in this release is optimized for cold data or write-once objects and files. It uses the zlib data compression library and favors saving space over speed. Usage on other types of data may result in performance degradation.
            • On Windows:
              • Compression of a file on Windows is only enabled via the mmchattr command.
              • The following Windows APIs are not supported:
                • FSCTL_SET_COMPRESSION to enable/disable compression on a file
                • FSCTL_GET_COMPRESSION to retrieve compression status of a file
                • The IBM zEDC hardware compression feature is enabled by default starting from IBM Spectrum Scale 5.1.0 when the CPU feature 'dflt' is listed in /proc/cpuinfo . Compression options such as z, zfast, alphae and alphah benefit from this feature. The feature also depends on your installed zlib version. For more information, see https://linux.mainframe.blog/zlib-acceleration/ .
                • Only four file system wide system classes are supported: maintenance , other , misc and mdio-sharing-class , while user class can be created based on user's request and can be used only after associating with a specific fileset.
                • User applications can be throttled differently by operating on different fileset.
                • The mmqos command is available only for the Linux operating system.
                • The QoS system classes cannot associate with filesets.
                • By default, the QoS system classes do not support MDIO.
                • For Linux on Z, Quality of Service is only supported with V4.2.1 and later.
                • No throttling for applications which perform direct I/O.
                • Not supported on AFM cache and AFM-based asynchronous DR filesets.
                • For IBM Spectrum Scale V4.2.1 and later, QoS is supported in an FPO environment.
                • The following flash for QoS was issued:
                  Abstract:
                  In an IBM Spectrum Scale V4.2 file system with multiple storage pools, Quality of Service (QoS) settings should be set for all storage pools to avoid performance degradation for unspecified storage pools.
                  Problem Summary:
                  In an IBM Spectrum Scale V4.2 file system with multiple storage pools, if the user specifies Quality of Service for I/O operations (QoS) settings (for the maintenance and other classes) only for one storage pool then the I/O allocations for the unspecified pools will be set to a very low value, resulting in severe performance degradation when I/O is performed to the unspecified storage pool(s).
                  See the complete Flash at http://www.ibm.com/support/docview.wss?uid=ssg1S1005464
                  The following considerations apply to running SELinux:

                  • From the 5.0.5 release, IBM Spectrum Scale runs on Red Hat Enterprise Linux operating systems with Security-Enhanced Linux (SELinux). For more information, see the Security-Enhanced Linux support topic in the IBM Spectrum Scale Concepts, Planning, and Installation Guide .
                  • When using the installation toolkit, the IBM Spectrum Scale Object protocol functionality requires the following SELinux packages to be installed:
                    • selinux-policy-base at 3.13.1-23 or higher
                    • selinux-policy-targeted at 3.12.1-153 or higher
                    • When using the Object protocol functionality, enabling SELinux after IBM Spectrum Scale has been installed is not supported. Contact IBM Spectrum Scale support at scale@us.ibm.com if you have questions about this restriction.
                    A2.32:
                    Enhancements to quota management available with IBM Spectrum Scale V4.1 and later, allow for quota clients to dynamically acquire and relinquish quotas based upon consuming rate; that is, the quota manager grants quota shares based upon global quota information such as the remaining quota limit and the number of mounted clients. Decisions based upon existing information provides for greater efficiency in managing quotas, however, this could result in an increase in the in-doubt values from earlier releases when running with a heavy IO workload. In order to get more accurate usage, issuing the mmlsquota and mmrepquota commands with the -e option might help.
                    Note: Quota report accuracy is mainly affected by hardware errors, such as node/network failures. Having a large number of nodes could increase the chances of having these types of failures.
                    The aggregate total number of quota records in user, group, and fileset quota files is limited to 200K records per file system. This limitation is due to the maximum amount of data that can be exchanged between the quota manager and quota command client such as the mmrepquota command.
                    Note: A large number of quota records per file system can result from the following scenarios:
                    • There are a very large number of users, groups, or filesets.
                    • If the --perfileset-quota option is enabled, the number of possible quota records is the number of filesets times number of users (and groups).

                    The immutability function of IBM Spectrum Scale 5.1.0 has been assessed for compliance in accordance to Securities and Exchange Commission (SEC) Rule 17a-4(f), Financial Industry Regulatory Authority (FINRA) Rule 4511(c) and the principles-based electronic records requirements of the Commodity Futures Trading Commission (CFTC) in 17 CFR § 1.31(c)-(d). To view the detailed assessment report, see IBM Spectrum Scale Assessment Report .

                    immutability function of IBM Spectrum Scale 5.0.0 has been assessed for compliance in accordance to US SEC17a-4f, EU GDPR Article 21 Section 1, German and Swiss laws and regulations by a recognized auditor. For more information, see the following links:

                    Assessment report: http://www.kpmg.de/bescheinigungen/RequestReport.aspx?B290411BE1224F5A9B4D24663BCD3C5D

                    Certificate: http://www.kpmg.de/bescheinigungen/RequestReport.aspx?DE968667B47544FF83F6CCDCF37E5FB5

                    A2.36:
                    Python 2 needs to be installed on new RHEL 8 installations because it is not installed by default. It is recommended that you create both the RHEL 8 BaseOS and AppStream repositories so that the package dependencies are met during installation.
                    If you intend to perform a leapp upgrade from RHEL 7.6 (versus a first-time installation) to RHEL 8, keep the following information in mind:
                    1. It is highly recommended that you use leapp-0.8.1-1 or higher. Make sure that the following requirements are met and the latest RHEL upgrade procedure is followed: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/upgrading_to_rhel_8/index
                    2. Consider running the boom utility to manage more boot loader entries on the system and to provide a path back to RHEL 7.6 if necessary: https://www.redhat.com/en/blog/upgrading-rhel-7-rhel-8-leapp-and-boom
                    3. As of the 5.0.4 release, if IBM Spectrum Scale 5.0.4 packages are installed and the leapp upgrade utility is run, leapp might remove some IBM Spectrum Scale packages and some dependencies. Manually reinstalling the removed packages will be required. Either proceed at your own risk, or consider not using the leapp upgrade utility and provisioning a new RHEL 8 node.
                    4. If protocols and authentication are enabled, python-ldap needs to be installed for mmadquery to run. A leapp upgrade from RHEL 7.x to RHEL 8.x removes the python-ldap package, which is a prerequisite for mmadquery . Ensure that you install python-ldap after the leapp upgrade if you intend to use mmadquery on RHEL 8.x.
                    Note:
                    RHEL 8 is not supported by the Object protocol in releases earlier than IBM Spectrum Scale 5.1.0.
                    RHEL 8 is supported by Transparent cloud tiering starting with IBM Spectrum Scale 5.0.5.
                    For more information, see Guidance for Red Hat Enterprise Linux 8.x on IBM Spectrum Scale nodes in the IBM Spectrum Scale: Concepts, Planning, and Installation Guide .
                    The following requirements and limitations apply to using the file clone function:
                    • A compressed file cannot be cloned and a clone file cannot be compressed.
                    • mmap is not supported on clone child files.

                    Starting with the IBM Spectrum Scale release 5.0.4, all IBM Spectrum Scale packages on Red Hat Enterprise Linux and SLES operating systems on supported architectures are signed by IBM with a GPG key. Starting with the IBM Spectrum Scale release 5.0.5.1, repository metadata is also signed by IBM.

                    You can use the available public key to verify the signatures on the packages and repository metadata.

                    The latest public key contents are as follows:
                    
                    -----BEGIN PGP PUBLIC KEY BLOCK-----
                    Version: EKM
                    mQENBF0tE4ABCADTU4imcpDlIHvcK/qWdMMrs72lL9EYDtA/JNL5YCPNeIa/54aIe3xXFJZbzkjs
                    v+5INaxYv0DEQxXEFq8vA1pQGPIG1elb3fXgP7Iyfiy13KDrVEB8AY/Cr/zTmHV8IJNMN8jcBl6Z
                    vAED7fXE82Q4jQ3djbg0OYBq2PeVS+wM5Y8n1+tmpVmcD9oLzYhJPeCsbFi6BAVgXBmyh4arrn15
                    OLSfD5jBnnOT926N2mpnsfubyGitQlywjJJuESnF9Ub9QMT7jNjGcg6frxHVOMUsIstmg01GBnvx
                    I/P/BvdiIqGjOTInka78+rYJpxZWPlbu/Xg/NXJ9sERjXuT30GCHABEBAAG0DVNwZWN0cnVtU2Nh
                    bGWJATkEEwEIACMFAl0tE4ACGy8HCwkIBwMCAQYVCAIJCgsEFgIDAQIeAQIXgAAKCRC9vnXD7+tu
                    654UCAClAR99Jhsdm47V2JvOBYLxcxdHqoqY+MqgKxeuy11Tp/enpqoGigZAcPbzlRvlJTOyh0Pa
                    PQC1y0oaKDROR5aOuCd0Cz3xbSQ92mWX0FkA7D9KNlAuxlGD6Ic58AvQ6RBv/mxblfH6gXHlc+Q0
                    +YFOY5YlvgKLYJ+exGngzieZfxspyyTab7FZe06G/lCm9U+mOfQ/7ODH6AvNIRmsCCg5uUeAmQOa
                    3+0RpWzN09nlkSYlkMlXvyZSWpwTEXLPtfDW0kzxsl1k4IzgyFMOsw6oLO5TVMZyL828MOuJtqU8
                    O6rS6+/RIho7GhiQ8SklugSFlFnT9fx5TRJcCiJmhHeF
                    =v7fu
                    -----END PGP PUBLIC KEY BLOCK-----
                    Save these contents in a file and import it to verify the signature manually. The installation toolkit does the signature verification automatically before installation or upgrade. For more information, see https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.5/com.ibm.spectrum.scale.v5r05.doc/bl1ins_verifysignature.htm .
                    Q2.40:
                    What are the components required for GPUDirect Storage (GDS) support with IBM Spectrum Scale?
                    A2.40:
                    The following software components are required for GPUDirect Storage (GDS) support with IBM Spectrum Scale:
                    • The required IBM Spectrum Scale version depends on the fabric type over which GPUDirect Storage is run:
                      • Infiniband: IBM Spectrum Scale 5.1.2 or later.
                      • RoCE: IBM Spectrum Scale 5.1.3 or later.
                      • The recommended version is IBM Spectrum Scale 5.1.6 as it includes accelerated GDS writes.
                      • For GDS clients: Ubuntu 20.04, RHEL 8.4, RHEL 8.6.
                      • For storage servers: Any Linux distro that supports IBM Spectrum Scale 5.1.2 (Infiniband), 5.1.3 (Infiniband, RoCE) or later.
                      • IBM Spectrum Scale 5.1.6 introduces accelerated GDS writes. For more information, see GPUDirect Storage support for IBM Spectrum Scale .
                      • Asynchronous CUDA IO is not supported.
                      • The IBM Spectrum Scale 5.1.1 technical preview is not compatible with IBM Spectrum Scale 5.1.2 and later.
                      References:
                      • For more information and supported hardware, see Planning for GPUDirectStorage in the IBM Spectrum Scale : Concepts, Planning, and Installation Guide .
                      • For the base configuration of IBM Spectrum Scale on RoCE fabrics, see Highly Efficient Data Access with RoCE on IBM Elastic Storage Systems and IBM Spectrum Scale .
                      Q2.41:
                      What are the requirements/limitations for manipulating the system.nfs4_acl extended attribute directly in IBM Spectrum Scale?
                      A2.41:
                      The following are the requirements/limitations for manipulating the system.nfs4_acl extended attribute directly in IBM Spectrum Scale:
                      • Manipulating the system.nfs4_acl extended attribute is supported only on Linux.
                      • When installing the nfs4-acl-tools package, it is recommended to use version 0.4.2 or later.
                      • nfs4-acl-tools 0.4.2 and later versions are expected to work without problems.
                      • nfs4-acl-tools 0.3.6, 0.3.7, and 0.4.1 have the problem of showing the error message Invalid filename for all the errors returned from stat() . For example, if a directory does not have execute permission for user A and user A invokes nfs4_getfacl on a file within the directory, Invalid filename is returned instead of Permission denied . Version 0.3.7 is included in Ubuntu 22.04.
                      • nfs4-acl-tools 0.3.5 has the problem of showing undocumented flags O, G, and E, which can be ignored. For example, if the parent directory has ACE A:d:gpfsuser:rtncy, a subdirectory created under it will inherit the ACE with the additional O flag A:do:gpfsuser:rtncy, the O flag is not part of the protocol and can be ignored. This version is included in RHEL 8 and RHEL 9.
                      • nfs4-acl-tools 0.3.3 can run into segmentation faults. This version is not supported. This is included in RHEL 7, Ubuntu 20.04, SLES15 SP3, and SLES15 SP4.
                      • Listing the extended attributes through listxattr does not include the system.nfs4_acl attribute. This is to avoid redundant work in case of preserving xattrs with cp ( cp --preserve=xattr ), as the system.gpfs_nfs4_acl extended attribute already exists. The system.nfs4_acl attribute can still be retrieved with getxattr .
                      • In addition to strings, numeric IDs can also be accepted for the principal in a NFSv4 access control entry.
                      • AIX on Power is supported by IBM POWER8 processors, or higher processors , supported by your level of AIX, with a minimum of 2 GB of system memory.
                      • Linux on Power is supported on IBM POWER8, or higher processors, with a minimum of 2 GB of system memory.
                      • POWER9 and Power10 CPUs support a new Radix MMU mode. The Linux kernel can use this MMU mode to implement additional restrictions for memory access, called Kernel Userspace Access Prevention (KUAP). IBM Spectrum Scale releases 5.1.3 and 5.1.2.5 have proper support for this feature. Earlier IBM Spectrum Scale releases require a workaround. The only affected Linux distribution by earlier IBM Spectrum Scale releases is Ubuntu 20.04. To verify whether the POWER9 and POWER10 feature is active, run the following command:

                        grep Radix /proc/cpuinfo

                        To check whether KUAP is active, run dmesg or check the syslog for the following message:

                        radix-mmu: Activating Kernel Userspace Access Prevention

                        If KUAP is active, modify the boot loader of your Linux distribution to pass the nosmap parameter to the Linux kernel. Then reboot and again run the above checks. Only when KUAP is no longer active, attempt to start IBM Spectrum Scale.

                        Note: For more information about specific Power processor and operating system version requirements, see 2.1 What is supported on IBM Spectrum Scale for AIX, Linux, Power, and Windows? .
                      • IBM Spectrum Scale V4/IBM GPFS V3 or later on x86 Architecture is supported on:
                      • Intel EM64T processors, with 2 GB of memory.
                      • AMD Opteron processors, with 2 GB of memory. Other AMD x86-64 processors are supported as long as they are completely compatible with AMD Opteron, and as long as the SMP scaling limit is not exceeded. For more information, see 5.3 What is the current maximum tested limit for SMP scaling? .
                      • Because of their sometimes unique communications fabric, or other elements of the system architecture, support of GPFS in any Cray system environment requires GPFS development review and approval. Contact scale@us.ibm.com to arrange for such a review.

                      Additionally, it is highly suggested that a sufficiently large amount of swap space is configured. While the actual configuration decisions should be made taking into account the memory requirements of other applications, it is suggested to configure at least as much swap space as there is physical memory on a given node.

                      IBM Spectrum Scale is supported on systems which are listed in, or compatible with, the IBM hardware specified in the Hardware requirements section of the Sales Manual for IBM Spectrum Scale.

                      To access the Sales Manual:
                      1. Go to http://www.ibm.com/common/ssi/index.wss
                      2. On Information Type, choose HW&SW Desc (sales manual,RPQ) .
                      3. For IBM Spectrum Scale V5, choose the corresponding product number to enter in the Search for field:
                         • IBM Spectrum Scale Data Access Edition: 5737-I39 (Passport Advantage), 5641-DA1 , 5641-DA3 , 5641-DA5 (eConfig/AAS)
                         • IBM Spectrum Scale Data Management Edition: 5737-F34 (Passport Advantage), 5641-DM1 , 5641-DM3 , 5641-DM5 (eConfig/AAS)
                         • IBM Spectrum Scale Erasure Code Edition: 5737J34 (Passport Advantage)
                         • IBM Spectrum Scale for Linux on Z is supported:
                           1. with the distributions and kernel levels as listed in the question 2.1 What is supported on IBM Spectrum Scale for AIX, Linux, Power, and Windows?
                           2. on servers that meet the minimum hardware model requirements as listed in the question What are the minimum hardware requirements for a IBM Spectrum Scale cluster?
                           3. with levels of AIX as listed in the question 2.1 What is supported on IBM Spectrum Scale for AIX, Linux, Power, and Windows?
                           4. on servers that meet the minimum hardware model requirements as listed in the question What are the minimum hardware requirements for a IBM Spectrum Scale cluster?
                              Note: IBM Spectrum Scale runs on the POWER8 processor in default, POWER8, and POWER7® compatibility modes.
                           5. IBM Spectrum Scale for Linux on Power is supported:
                              1. with the distributions and kernel levels as listed in the question 2.1 What is supported on IBM Spectrum Scale for AIX, Linux, Power, and Windows?
                              2. on servers that meet the minimum hardware model requirements as listed in the question What are the minimum hardware requirements for a IBM Spectrum Scale cluster?
                                 Note: IBM Spectrum Scale runs on the POWER8 processor in default, POWER8, and POWER7 compatibility modes.
                              3. IBM Spectrum Scale for Linux on x86 Architecture is supported:
                                 1. with the distributions and kernel levels as listed in the question 2.1 What is supported on IBM Spectrum Scale for AIX, Linux, Power, and Windows?
                                 2. on servers that meet the minimum hardware model requirements as listed in the question What are the minimum hardware requirements for a IBM Spectrum Scale cluster?
                                 3. for additional support statements, see the question What are the current restrictions on IBM Spectrum Scale Linux kernel support?
                                 4. IBM Spectrum Scale for Windows on x86 Architecture is supported:
                                 5. with the levels of Windows Server as listed in the question 2.1 What is supported on IBM Spectrum Scale for AIX, Linux, Power, and Windows?
                                 6. on servers that meet the minimum hardware model requirements as listed in the question What are the minimum hardware requirements for a IBM Spectrum Scale cluster?
                                 What interconnects are supported for GPFS daemon-to-daemon communication in a GPFS cluster?
                                 A3.3:
                                 The interconnect for GPFS daemon-to-daemon communication depends upon the types of nodes in your cluster.
                                 Note: This table provides the list of communication interconnects which have been tested by IBM and are known to work with GPFS. Other interconnects may work with GPFS but they have not been tested by IBM. The IBM Spectrum Scale support team will help customers who are using interconnects that have not been tested to solve problems directly related to GPFS, but will not be responsible for solving problems deemed to be issues with the underlying communication interconnect's behavior including any performance issues exhibited on untested interconnects.