cudnnStatus_t cudnnBuildRNNDynamic(
    cudnnHandle_t handle,
    cudnnRNNDescriptor_t rnnDesc,
    int32_t miniBatch);
Parameters
handle
Input. Handle to a previously created cuDNN context.
rnnDesc
Input. A previously initialized RNN descriptor.
miniBatch
Input. The exact number of sequences in a batch.
Returns
CUDNN_STATUS_SUCCESS
The code was built and linked successfully.
CUDNN_STATUS_MAPPING_ERROR
A GPU/CUDA resource, such as a texture object, shared memory, or zero-copy memory is not available in the required size or there is a mismatch between the user resource and cuDNN internal resources. A resource mismatch may occur, for example, when calling cudnnSetStream(). There could be a mismatch between the user provided CUDA stream and the internal CUDA events instantiated in the cuDNN handle when cudnnCreate() was invoked.
This error status may not be correctable when it is related to texture dimensions, shared memory size, or zero-copy memory availability. If CUDNN_STATUS_MAPPING_ERROR is returned by cudnnSetStream(), then it is typically correctable, however, it means that the cuDNN handle was created on one GPU and the user stream passed to this function is associated with another GPU.
CUDNN_STATUS_ALLOC_FAILED
The resources could not be allocated.
CUDNN_STATUS_RUNTIME_PREREQUISITE_MISSING
The prerequisite runtime library could not be found.
CUDNN_STATUS_NOT_SUPPORTED
The current hyper-parameters are invalid.
cudnnCreateAttnDescriptor()#
This function has been deprecated in cuDNN 9.0.
This function creates one instance of an opaque attention descriptor object by allocating the host memory for it and initializing all descriptor fields.  The function writes NULL to attnDesc when the attention descriptor object cannot be allocated.
cudnnStatus_t cudnnCreateAttnDescriptor(cudnnAttnDescriptor_t *attnDesc);
Use the cudnnSetAttnDescriptor() function to configure the attention descriptor and cudnnDestroyAttnDescriptor() to destroy it and release the allocated memory.
Parameters
attnDesc
Output. Pointer where the address to the newly created attention descriptor should be written.
Returns
CUDNN_STATUS_SUCCESS
The descriptor object was created successfully.
CUDNN_STATUS_BAD_PARAM
An invalid input argument was encountered (attnDesc=NULL).
CUDNN_STATUS_ALLOC_FAILED
The memory allocation failed.
This function creates a CTC loss function descriptor.
cudnnStatus_t cudnnCreateCTCLossDescriptor(
    cudnnCTCLossDescriptor_t* ctcLossDesc)
Parameters
ctcLossDesc
Output. CTC loss descriptor to be set. For more information, refer to cudnnCTCLossDescriptor_t.
Returns
CUDNN_STATUS_SUCCESS
The function returned successfully.
CUDNN_STATUS_BAD_PARAM
The CTC loss descriptor passed to the function is invalid.
CUDNN_STATUS_ALLOC_FAILED
Memory allocation for this CTC loss descriptor failed.
cudnnCreateRNNDataDescriptor()#
This function creates a RNN data descriptor object by allocating the memory needed to hold its opaque structure.
cudnnStatus_t cudnnCreateRNNDataDescriptor(
    cudnnRNNDataDescriptor_t *RNNDataDesc)
Parameters
RNNDataDesc
Output. Pointer to where the address to the newly created RNN data descriptor should be written.




    

Returns
CUDNN_STATUS_SUCCESS
The RNN data descriptor object was created successfully.
CUDNN_STATUS_BAD_PARAM
The RNNDataDesc argument is NULL.
CUDNN_STATUS_ALLOC_FAILED
The resources could not be allocated.
cudnnCreateRNNDescriptor()#
This function creates a generic RNN descriptor object by allocating the memory needed to hold its opaque structure.
cudnnStatus_t cudnnCreateRNNDescriptor(
    cudnnRNNDescriptor_t    *rnnDesc)
Parameters
rnnDesc
Output. Pointer to where the address to the newly created RNN descriptor should be written.
Returns
CUDNN_STATUS_SUCCESS
The object was created successfully.
CUDNN_STATUS_BAD_PARAM
The rnnDesc argument is NULL.
CUDNN_STATUS_ALLOC_FAILED
The resources could not be allocated.
cudnnCreateSeqDataDescriptor()#
This function has been deprecated in cuDNN 9.0.
This function creates one instance of an opaque sequence data descriptor object by allocating the host memory for it and initializing all descriptor fields. The function writes NULL to seqDataDesc when the sequence data descriptor object cannot be allocated.
cudnnStatus_t cudnnCreateSeqDataDescriptor(cudnnSeqDataDescriptor_t *seqDataDesc)
Use the cudnnSetSeqDataDescriptor() function to configure the sequence data descriptor and cudnnDestroySeqDataDescriptor() to destroy it and release the allocated memory.
Parameters
seqDataDesc
Output. Pointer where the address to the newly created sequence data descriptor should be written.
Returns
CUDNN_STATUS_SUCCESS
The descriptor object was created successfully.
CUDNN_STATUS_BAD_PARAM
An invalid input argument was encountered (seqDataDesc=NULL).
CUDNN_STATUS_ALLOC_FAILED
The memory allocation failed.
cudnnCTCLoss()#
This function returns the CTC costs and gradients, given the probabilities and labels.
cudnnStatus_t cudnnCTCLoss(
    cudnnHandle_t                        handle,
    const   cudnnTensorDescriptor_t      probsDesc,
    const   void                        *probs,
    const   int                          hostLabels[],
    const   int                          hostLabelLengths[],
    const   int                          hostInputLengths[],
    void                                *costs,
    const   cudnnTensorDescriptor_t      gradientsDesc,
    const   void                        *gradients,
    cudnnCTCLossAlgo_t                   algo,
    const   cudnnCTCLossDescriptor_t     ctcLossDesc,
    void                                *workspace,
    size_t                              *workSpaceSizeInBytes)
This function can have an inconsistent interface depending on the cudnnLossNormalizationMode_t chosen (bound to the cudnnCTCLossDescriptor_t with cudnnSetCTCLossDescriptorEx()). For the CUDNN_LOSS_NORMALIZATION_NONE, this function has an inconsistent interface, for example, the probs input is probability normalized by softmax, but the gradients output is with respect to the unnormalized activation. However, for CUDNN_LOSS_NORMALIZATION_SOFTMAX, the function has a consistent interface; all values are normalized by softmax.
Parameters
handle
Input. Handle to a previously created cuDNN context. For more information, refer to cudnnHandle_t.
probsDesc
Input. Handle to the previously initialized probabilities tensor descriptor. For more information, refer to cudnnTensorDescriptor_t.
probs
Input. Pointer to a previously initialized probabilities tensor. These input probabilities are normalized by softmax.
hostLabels
Input. Pointer to a previously initialized labels list, in CPU memory.
hostLabelLengths
Input. Pointer to a previously initialized lengths list in CPU memory, to walk the above labels list.
hostInputLengths
Input. Pointer to a previously initialized list of the lengths of the timing steps in each batch, in CPU memory.
costs
Output. Pointer to the computed costs of CTC.
gradientsDesc
Input. Handle to a previously initialized gradient tensor descriptor.
gradients
Output. Pointer to the computed gradients of CTC. These computed gradient outputs are with respect to the unnormalized activation.
algo
Input. Enumerant that specifies the chosen CTC loss algorithm. For more information, refer to cudnnCTCLossAlgo_t.
ctcLossDesc
Input. Handle to the previously initialized CTC loss descriptor. For more information, refer to cudnnCTCLossDescriptor_t.
workspace
Input. Pointer to GPU memory of a workspace needed to be able to execute the specified algorithm.
sizeInBytes
Input. Amount of GPU memory needed as workspace to be able to execute the CTC loss computation with the specified algo.
Returns
CUDNN_STATUS_SUCCESS
The query was successful.
CUDNN_STATUS_BAD_PARAM
At least one of the following conditions are met:
The dimensions of probsDesc do not match the dimensions of gradientsDesc.
The inputLengths do not agree with the first dimension of probsDesc.
The workSpaceSizeInBytes is not sufficient.
The labelLengths is greater than 255.
CUDNN_STATUS_NOT_SUPPORTED
A compute or data type other than FLOAT was chosen, or an unknown algorithm type was chosen.
CUDNN_STATUS_EXECUTION_FAILED
The function failed to launch on the GPU.
cudnnCTCLoss_v8()#
This function returns the CTC costs and gradients, given the probabilities and labels. Many CTC API functions were updated in version 8 with the _v8 suffix to support CUDA graphs. Label and input data is now passed in GPU memory.
cudnnStatus_t cudnnCTCLoss_v8(
    cudnnHandle_t                        handle,
    cudnnCTCLossAlgo_t                   algo,
    const   cudnnCTCLossDescriptor_t     ctcLossDesc,
    const   cudnnTensorDescriptor_t      probsDesc,
    const   void                        *probs,
    const   int                          labels[],
    const   int                          labelLengths[],
    const   int                          inputLengths[],
    void                                *costs,
    const   cudnnTensorDescriptor_t      gradientsDesc,
    const   void                        *gradients,
    size_t                              *workSpaceSizeInBytes,
    void                                *workspace)
This function can have an inconsistent interface depending on the cudnnLossNormalizationMode_t chosen (bound to the cudnnCTCLossDescriptor_t with cudnnSetCTCLossDescriptorEx()). For the CUDNN_LOSS_NORMALIZATION_NONE, this function has an inconsistent interface, for example, the probs input is probability normalized by softmax, but the gradients output is with respect to the unnormalized activation. However, for CUDNN_LOSS_NORMALIZATION_SOFTMAX, the function has a consistent interface; all values are normalized by softmax.
Parameters
handle
Input. Handle to a previously created cuDNN context. For more information, refer to cudnnHandle_t




    
.
algo
Input. Enumerant that specifies the chosen CTC loss algorithm. For more information, refer to cudnnCTCLossAlgo_t.
ctcLossDesc
Input. Handle to the previously initialized CTC loss descriptor. For more information, refer to cudnnCTCLossDescriptor_t.
probsDesc
Input. Handle to the previously initialized probabilities tensor descriptor. For more information, refer to cudnnTensorDescriptor_t.
probs
Input. Pointer to a previously initialized probabilities tensor. These input probabilities are normalized by softmax.
labels
Input. Pointer to a previously initialized labels list, in GPU memory.
labelLengths
Input. Pointer to a previously initialized lengths list in GPU memory, to walk the above labels list.
inputLengths
Input. Pointer to a previously initialized list of the lengths of the timing steps in each batch, in GPU memory.
costs
Output. Pointer to the computed costs of CTC.
gradientsDesc
Input. Handle to a previously initialized gradient tensor descriptor.
gradients
Output. Pointer to the computed gradients of CTC. These computed gradient outputs are with respect to the unnormalized activation.
workspace
Input. Pointer to GPU memory of a workspace needed to be able to execute the specified algorithm.
sizeInBytes
Input. Amount of GPU memory needed as a workspace to be able to execute the CTC loss computation with the specified algo.
Returns
CUDNN_STATUS_SUCCESS
The query was successful.
CUDNN_STATUS_BAD_PARAM
At least one of the following conditions are met:
The dimensions of probsDesc do not match the dimensions of gradientsDesc.
The workSpaceSizeInBytes is not sufficient.
CUDNN_STATUS_NOT_SUPPORTED
A compute or data type other than FLOAT was chosen, or an unknown algorithm type was chosen.
CUDNN_STATUS_EXECUTION_FAILED
The function failed to launch on the GPU.
cudnnDestroyAttnDescriptor()#
This function has been deprecated in cuDNN 9.0.
This function destroys the attention descriptor object and releases its memory. The attnDesc argument can be NULL. Invoking cudnnDestroyAttnDescriptor() with a NULL argument is a no operation (NOP).
cudnnStatus_t cudnnDestroyAttnDescriptor(cudnnAttnDescriptor_t attnDesc);
The cudnnDestroyAttnDescriptor() function is not able to detect if the attnDesc argument holds a valid address. Undefined behavior will occur in case of passing an invalid pointer, not returned by the cudnnCreateAttnDescriptor() function, or in the double deletion scenario of a valid address.
Parameters
attnDesc
Input. Pointer to the attention descriptor object to be destroyed.
Returns
CUDNN_STATUS_SUCCESS
The descriptor was destroyed successfully.
cudnnDestroyCTCLossDescriptor()#
This function destroys a CTC loss function descriptor object.
cudnnStatus_t cudnnDestroyCTCLossDescriptor(
    cudnnCTCLossDescriptor_t    ctcLossDesc)
Parameters
ctcLossDesc
Input. CTC loss function descriptor to be destroyed.
Returns
CUDNN_STATUS_SUCCESS
The function returned successfully.
cudnnDestroyRNNDataDescriptor()#
This function destroys a previously created RNN data descriptor object. Invoking cudnnDestroyRNNDataDescriptor() with the NULL argument is a no operation (NOP).
cudnnStatus_t cudnnDestroyRNNDataDescriptor(
    cudnnRNNDataDescriptor_t RNNDataDesc)
The cudnnDestroyRNNDataDescriptor() function is not able to detect if the RNNDataDesc argument holds a valid address. Undefined behavior will occur in cases of passing an invalid pointer, not returned by the cudnnCreateRNNDataDescriptor() function, or in the double deletion scenario of a valid address.
Parameters
RNNDataDesc
Input. Pointer to the RNN data descriptor object to be destroyed.
Returns
CUDNN_STATUS_SUCCESS
The RNN data descriptor object was destroyed successfully.
cudnnDestroyRNNDescriptor()#
This function destroys a previously created RNN descriptor object. Invoking cudnnDestroyRNNDescriptor() with the NULL argument is a no operation (NOP).
cudnnStatus_t cudnnDestroyRNNDescriptor(
    cudnnRNNDescriptor_t rnnDesc)
The cudnnDestroyRNNDescriptor() function is not able to detect if the rnnDesc argument holds a valid address. Undefined behavior will occur in cases of passing an invalid pointer, not returned by the cudnnCreateRNNDescriptor() function, or in the double deletion scenario of a valid address.
Parameters
rnnDesc
Input. Pointer to the RNN descriptor object to be destroyed.
Returns
CUDNN_STATUS_SUCCESS
The object was destroyed successfully.
cudnnDestroySeqDataDescriptor()#
This function has been deprecated in cuDNN 9.0.
This function destroys the sequence data descriptor object and releases its memory. The seqDataDesc argument can be NULL. Invoking cudnnDestroySeqDataDescriptor() with a NULL argument is a no operation (NOP).
cudnnStatus_t cudnnDestroySeqDataDescriptor(cudnnSeqDataDescriptor_t seqDataDesc);
The cudnnDestroySeqDataDescriptor() function is not able to detect if the seqDataDesc argument holds a valid address. Undefined behavior will occur in case of passing an invalid pointer, not returned by the cudnnCreateSeqDataDescriptor() function, or in the double deletion scenario of a valid address.
Parameters
seqDataDesc
Input. Pointer to the sequence data descriptor object to be destroyed.
Returns
CUDNN_STATUS_SUCCESS
The descriptor was destroyed successfully.
cudnnGetAttnDescriptor()#
This function has been deprecated in cuDNN 9.0.
This function retrieves settings from the previously created attention descriptor. The user can assign NULL to any pointer except attnDesc when the retrieved value is not needed.
cudnnStatus_t cudnnGetAttnDescriptor(
    cudnnAttnDescriptor_t attnDesc,
    unsigned *attnMode,
    int *nHeads,
    double *smScaler,
    cudnnDataType_t *dataType,
    cudnnDataType_t *computePrec,
    cudnnMathType_t *mathType,
    cudnnDropoutDescriptor_t *attnDropoutDesc,
    cudnnDropoutDescriptor_t *postDropoutDesc,
    int *qSize,
    int *kSize,
    int *vSize,
    int *qProjSize,
    int *kProjSize,
    int *vProjSize,
    int *oProjSize,
    int *qoMaxSeqLength,
    int *kvMaxSeqLength,
    int *maxBatchSize,
    int *maxBeamSize);
Parameters
attnDesc
Input. Attention descriptor.
attnMode
Output. Pointer to the storage for binary attention flags.
nHeads
Output. Pointer to the storage for the number of attention heads.
smScaler
Output. Pointer to the storage for the softmax smoothing/sharpening coefficient.
dataType
Output. Data type for attention weights, sequence data inputs, and outputs.
computePrec




    
Output. Pointer to the storage for the compute precision.
mathType
Output. NVIDIA Tensor Core settings.
attnDropoutDesc
Output. Descriptor of the dropout operation applied to the softmax output.
postDropoutDesc
Output. Descriptor of the dropout operation applied to the multihead attention output.
qSize, kSize, vSize
Output. Q, K, and V embedding vector lengths.
qProjSize, kProjSize, vProjSize
Output. Q, K, and V embedding vector lengths after input projections.
oProjSize
Output. Pointer to store the output vector length after projection.
qoMaxSeqLength
Output. Largest sequence length expected in sequence data descriptors related to Q, O, dQ, dO inputs and outputs.
kvMaxSeqLength
Output. Largest sequence length expected in sequence data descriptors related to K, V, dK, dV inputs and outputs.
maxBatchSize
Output. Largest batch size expected in the cudnnSeqDataDescriptor_t container.
maxBeamSize
Output. Largest beam size expected in the cudnnSeqDataDescriptor_t container.
Returns
CUDNN_STATUS_SUCCESS
Requested attention descriptor fields were retrieved successfully.
CUDNN_STATUS_BAD_PARAM
An invalid input argument was found.
cudnnGetCTCLossDescriptor()#
This function has been deprecated in cuDNN 9.0; use cudnnGetCTCLossDescriptor_v9() instead.
This function returns the configuration of the passed CTC loss function descriptor.
cudnnStatus_t cudnnGetCTCLossDescriptor(
    cudnnCTCLossDescriptor_t         ctcLossDesc,
    cudnnDataType_t*                 compType)
Parameters
ctcLossDesc
Input. CTC loss function descriptor passed, from which to retrieve the configuration.
compType
Output. Compute type associated with this CTC loss function descriptor.
Returns
CUDNN_STATUS_SUCCESS
The function returned successfully.
CUDNN_STATUS_BAD_PARAM
Input ctcLossDesc descriptor passed is invalid.
cudnnGetCTCLossDescriptor_v8()#
This function has been deprecated in cuDNN 9.0; use cudnnGetCTCLossDescriptor_v9() instead.
This function returns the configuration of the passed CTC loss function descriptor.
cudnnStatus_t cudnnGetCTCLossDescriptor_v8(
    cudnnCTCLossDescriptor_t         ctcLossDesc,
    cudnnDataType_t                 *compType,
    cudnnLossNormalizationMode_t    *normMode,
    cudnnNanPropagation_t           *gradMode,
    int                             *maxLabelLength)
Parameters
ctcLossDesc
Input. CTC loss function descriptor passed, from which to retrieve the configuration.
compType
Output. Compute type associated with this CTC loss function descriptor.
normMode
Output. Input normalization type for this CTC loss function descriptor. For more information, refer to cudnnLossNormalizationMode_t.
gradMode
Output. NaN propagation type for this CTC loss function descriptor.
maxLabelLength
Output. The max label length for this CTC loss function descriptor.
Returns
CUDNN_STATUS_SUCCESS
The function returned successfully.
CUDNN_STATUS_BAD_PARAM
Input ctcLossDesc descriptor passed is invalid.
cudnnGetCTCLossDescriptor_v9()#
This function returns the configuration of the passed CTC loss function descriptor.
cudnnStatus_t cudnnGetCTCLossDescriptor_v8(
    cudnnCTCLossDescriptor_t         ctcLossDesc,
    cudnnDataType_t                 *compType,
    cudnnLossNormalizationMode_t    *normMode,
    cudnnCTCGradMode_t              *ctcGradMode,
    int                             *maxLabelLength)
Parameters
ctcLossDesc
Input. CTC loss function descriptor passed, from which to retrieve the configuration.
compType
Output. Compute type associated with this CTC loss function descriptor.
normMode
Output. Input normalization type for this CTC loss function descriptor. For more information, refer to cudnnLossNormalizationMode_t.
ctcGradMode
Output. The gradient mode for handling OOB samples for this CTC loss function descriptor. Refer to cudnnSetCTCLossDescriptor_v9() for more information.
maxLabelLength
Output. The max label length for this CTC loss function descriptor.
Returns
CUDNN_STATUS_SUCCESS
The function returned successfully.
CUDNN_STATUS_BAD_PARAM
Input ctcLossDesc descriptor passed is invalid.
cudnnGetCTCLossDescriptorEx()#
This function has been deprecated in cuDNN 9.0; use cudnnGetCTCLossDescriptor_v9() instead.
This function returns the configuration of the passed CTC loss function descriptor.
cudnnStatus_t cudnnGetCTCLossDescriptorEx(
    cudnnCTCLossDescriptor_t         ctcLossDesc,
    cudnnDataType_t                 *compType,
    cudnnLossNormalizationMode_t    *normMode,
    cudnnNanPropagation_t           *gradMode)
Parameters
ctcLossDesc
Input. CTC loss function descriptor passed, from which to retrieve the configuration.
compType
Output. Compute type associated with this CTC loss function descriptor.
normMode
Output. Input normalization type for this CTC loss function descriptor. For more information, refer to cudnnLossNormalizationMode_t.
gradMode
Output. NaN propagation type for this CTC loss function descriptor.
Returns
CUDNN_STATUS_SUCCESS
The function returned successfully.
CUDNN_STATUS_BAD_PARAM
Input ctcLossDesc descriptor passed is invalid.
cudnnGetCTCLossWorkspaceSize()#
This function returns the amount of GPU memory workspace the user needs to allocate to be able to call cudnnCTCLoss() with the specified algorithm. The workspace allocated will then be passed to the routine cudnnCTCLoss().
cudnnStatus_t cudnnGetCTCLossWorkspaceSize(
    cudnnHandle_t                        handle,
    const   cudnnTensorDescriptor_t      probsDesc,
    const   cudnnTensorDescriptor_t      gradientsDesc,
    const   int                         *labels,
    const   int                         *labelLengths,
    const   int                         *inputLengths,
    cudnnCTCLossAlgo_t                   algo,
    const   cudnnCTCLossDescriptor_t     ctcLossDesc,
    size_t                              *sizeInBytes)
Parameters
handle
Input. Handle to a previously created cuDNN context.
probsDesc
Input. Handle to the previously initialized probabilities tensor descriptor.
gradientsDesc
Input. Handle to a previously initialized gradient tensor descriptor.
labels
Input. Pointer to a previously initialized labels list.
labelLengths
Input. Pointer to a previously initialized lengths list, to walk the above labels list.
inputLengths
Input. Pointer to a previously initialized list of the lengths of the timing steps in each batch.
algo
Input. Enumerant that specifies the chosen CTC loss algorithm.




    

ctcLossDesc
Input. Handle to the previously initialized CTC loss descriptor.
sizeInBytes
Output. Amount of GPU memory needed as workspace to be able to execute the CTC loss computation with the specified algo.
Returns
CUDNN_STATUS_SUCCESS
The query was successful.
CUDNN_STATUS_BAD_PARAM
At least one of the following conditions are met:
The dimensions of probsDesc do not match the dimensions of gradientsDesc
The inputLengths do not agree with the first dimension of probsDesc
The workSpaceSizeInBytes is not sufficient
The labelLengths is greater than 256
CUDNN_STATUS_NOT_SUPPORTED
A compute or data type other than FLOAT was chosen, or an unknown algorithm type was chosen.
cudnnGetCTCLossWorkspaceSize_v8()#
This function returns the amount of GPU memory workspace the user needs to allocate to be able to call cudnnCTCLoss_v8 with the specified algorithm. The workspace allocated will then be passed to the routine cudnnCTCLoss_v8().
cudnnStatus_t cudnnGetCTCLossWorkspaceSize_v8(
    cudnnHandle_t                        handle,
    cudnnCTCLossAlgo_t                   algo,
    const   cudnnCTCLossDescriptor_t     ctcLossDesc,
    const   cudnnTensorDescriptor_t      probsDesc,
    const   cudnnTensorDescriptor_t      gradientsDesc,
    size_t                              *sizeInBytes)
Parameters
handle
Input. Handle to a previously created cuDNN context.
algo
Input. Enumerant that specifies the chosen CTC loss algorithm.
ctcLossDesc
Input. Handle to the previously initialized CTC loss descriptor.
probsDesc
Input. Handle to the previously initialized probabilities tensor descriptor.
gradientsDesc
Input. Handle to a previously initialized gradient tensor descriptor.
sizeInBytes
Output. Amount of GPU memory needed as workspace to be able to execute the CTC loss computation with the specified algo.
Returns
CUDNN_STATUS_SUCCESS
The query was successful.
CUDNN_STATUS_BAD_PARAM
At least one of the following conditions are met:
The dimensions of probsDesc do not match the dimensions of gradientsDesc
CUDNN_STATUS_NOT_SUPPORTED
- A compute or data type other than FLOAT was chosen, or an unknown algorithm type was chosen.
- For the deterministic CTC loss algorithm, the maxLabelLength in ctcLossDesc is greater than or equal to 256.
- For the nondeterministic CTC loss algorithm, the maxLabelLength in ctcLossDesc is greater than or equal to 2048.
cudnnGetMultiHeadAttnBuffers()#
This function has been deprecated in cuDNN 9.0.
This function computes weight, work, and reserve space buffer sizes used by the following functions:
cudnnMultiHeadAttnForward()
cudnnMultiHeadAttnBackwardData()
cudnnMultiHeadAttnBackwardWeights()
cudnnStatus_t cudnnGetMultiHeadAttnBuffers(
    cudnnHandle_t handle,
    const cudnnAttnDescriptor_t attnDesc,
    size_t *weightSizeInBytes,
    size_t *workSpaceSizeInBytes,
    size_t *reserveSpaceSizeInBytes);
Assigning NULL to the reserveSpaceSizeInBytes argument indicates that the user does not plan to invoke multihead attention gradient functions: cudnnMultiHeadAttnBackwardData() and cudnnMultiHeadAttnBackwardWeights(). This situation occurs in the inference mode.
NULL cannot be assigned to weightSizeInBytes and workSpaceSizeInBytes pointers.
The user must allocate weight, work, and reserve space buffer sizes in the GPU memory using cudaMalloc() with the reported buffer sizes. The buffers can be also carved out from a larger chunk of allocated memory but the buffer addresses must be at least 16B aligned.
The workspace buffer is used for temporary storage. Its content can be discarded or modified after all GPU kernels launched by the corresponding API complete.  The reserve-space buffer is used to transfer intermediate results from cudnnMultiHeadAttnForward() to cudnnMultiHeadAttnBackwardData(), and from cudnnMultiHeadAttnBackwardData() to cudnnMultiHeadAttnBackwardWeights(). The content of the reserve-space buffer cannot be modified until all GPU kernels launched by the above three multihead attention API functions finish.
All multihead attention weight and bias tensors are stored in a single weight buffer. For speed optimizations, the cuDNN API may change tensor layouts and their relative locations in the weight buffer based on the provided attention parameters. Use the cudnnGetMultiHeadAttnWeights() function to obtain the start address and the shape of each weight or bias tensor.
Parameters
handle
Input. The current cuDNN context handle.
attnDesc
Input. Pointer to a previously initialized attention descriptor.
weightSizeInBytes
Output. Minimum buffer size required to store all multihead attention trainable parameters.
workSpaceSizeInBytes
Output. Minimum buffer size required to hold all temporary surfaces used by the forward and gradient multihead attention API calls.
reserveSpaceSizeInBytes
Output. Minimum buffer size required to store all intermediate data exchanged between forward and backward (gradient) multihead attention functions. Set this parameter to NULL in the inference mode indicating that gradient API calls will not be invoked.
Returns
CUDNN_STATUS_SUCCESS
The requested buffer sizes were computed successfully.
CUDNN_STATUS_BAD_PARAM
An invalid input argument was found.
cudnnGetMultiHeadAttnWeights()#
This function has been deprecated in cuDNN 9.0.
This function obtains the shape of the weight or bias tensor. It also retrieves the start address of tensor data located in the weight buffer.  Use the wKind argument to select a particular tensor.  For more information, refer to cudnnMultiHeadAttnWeightKind_t for the description of the enumerant type.
cudnnStatus_t cudnnGetMultiHeadAttnWeights(
    cudnnHandle_t handle,
    const cudnnAttnDescriptor_t attnDesc,
    cudnnMultiHeadAttnWeightKind_t wKind,
    size_t weightSizeInBytes,
    const void *weights,
    cudnnTensorDescriptor_t wDesc,
    void **wAddr);
Biases are used in the input and output projections when the CUDNN_ATTN_ENABLE_PROJ_BIASES flag is set in the attention descriptor. Refer to cudnnSetAttnDescriptor() for the description of flags to control projection biases.
When the corresponding weight or bias tensor does not exist, the function writes NULL to the storage location pointed by wAddr and returns zeros in the wDesc tensor descriptor.  The return status of the cudnnGetMultiHeadAttnWeights() function is CUDNN_STATUS_SUCCESS in this case.
The cuDNN multiHeadAttention sample code demonstrates how to access multihead attention weights.  Although the buffer with weights and biases should be allocated in the GPU memory, the user can copy it to the host memory and invoke the cudnnGetMultiHeadAttnWeights() function with the host weights address to obtain tensor pointers in the host memory.  This scheme allows the user to inspect trainable parameters directly in the CPU memory.
Parameters
handle
Input. The current cuDNN context handle.
attnDesc
Input. A previously configured attention descriptor.
wKind
Input. Enumerant type to specify which weight or bias tensor should be retrieved.
weightSizeInBytes
Input. Buffer size that stores all multihead attention weights and biases.
weights
Input. Pointer to the weight buffer in the host or device memory.
wDesc
Output. The descriptor specifying weight or bias tensor shape. For weights, the wDesc.dimA[] array has three elements: [nHeads, projected size, original size].  For biases, the wDesc.dimA[]




    
 array also has three elements: [nHeads, projected size, 1].  The wDesc.strideA[] array describes how tensor elements are arranged in memory.
wAddr
Output. Pointer to a location where the start address of the requested tensor should be written.  When the corresponding projection is disabled, the address written to wAddr is NULL.
Returns
CUDNN_STATUS_SUCCESS
The weight tensor descriptor and the address of data in the device memory were successfully retrieved.
CUDNN_STATUS_BAD_PARAM
An invalid or incompatible input argument was encountered. For example, wKind did not have a valid value or weightSizeInBytes was too small.
cudnnGetRNNDataDescriptor()#
This function retrieves a previously created RNN data descriptor object.
cudnnStatus_t cudnnGetRNNDataDescriptor(
    cudnnRNNDataDescriptor_t       RNNDataDesc,
    cudnnDataType_t                *dataType,
    cudnnRNNDataLayout_t           *layout,
    int                            *maxSeqLength,
    int                            *batchSize,
    int                            *vectorSize,
    int                            arrayLengthRequested,
    int                            seqLengthArray[],
    void                           *paddingFill);
Parameters
RNNDataDesc
Input. A previously created and initialized RNN descriptor.
dataType
Output. Pointer to the host memory location to store the datatype of the RNN data tensor.
layout
Output. Pointer to the host memory location to store the memory layout of the RNN data tensor.
maxSeqLength
Output. The maximum sequence length within this RNN data tensor, including the padding vectors.
batchSize
Output. The number of sequences within the mini-batch.
vectorSize
Output. The vector length (meaning, embedding size) of the input or output tensor at each time-step.
arrayLengthRequested
Input. The number of elements that the user requested for seqLengthArray.
seqLengthArray
Output. Pointer to the host memory location to store the integer array describing the length (meaning, number of timesteps) of each sequence. This is allowed to be a NULL pointer if arrayLengthRequested is 0.
paddingFill
Output. Pointer to the host memory location to store the user defined symbol. The symbol should be interpreted as the same data type as the RNN data tensor.
Returns
CUDNN_STATUS_SUCCESS
The parameters are fetched successfully.
CUDNN_STATUS_BAD_PARAM
Any one of these have occurred:
Any of RNNDataDesc, dataType, layout, maxSeqLength, batchSize, vectorSize, or paddingFill is NULL.
seqLengthArray is NULL while arrayLengthRequested is greater than zero.
arrayLengthRequested is less than zero.
cudnnGetRNNDescriptor_v8()#
This function retrieves RNN network parameters that were configured by cudnnSetRNNDescriptor_v8(). The user can assign NULL to any pointer except rnnDesc when the retrieved value is not needed. The function does not check the validity of retrieved parameters.
cudnnStatus_t cudnnGetRNNDescriptor_v8(
    cudnnRNNDescriptor_t rnnDesc,
    cudnnRNNAlgo_t *algo,
    cudnnRNNMode_t *cellMode,
    cudnnRNNBiasMode_t *biasMode,
    cudnnDirectionMode_t *dirMode,
    cudnnRNNInputMode_t *inputMode,
    cudnnDataType_t *dataType,
    cudnnDataType_t *mathPrec,
    cudnnMathType_t *mathType,
    int32_t *inputSize,
    int32_t *hiddenSize,
    int32_t *projSize,
    int32_t *numLayers,
    cudnnDropoutDescriptor_t *dropoutDesc,
    uint32_t *auxFlags);
Parameters
rnnDesc
Input. A previously created and initialized RNN descriptor.
algo
Output. Pointer to where RNN algorithm type should be stored.
cellMode
Output. Pointer to where RNN cell type should be saved.
biasMode
Output. Pointer to where RNN bias mode cudnnRNNBiasMode_t should be saved.
dirMode
Output. Pointer to where RNN unidirectional/bidirectional mode should be saved.
inputMode
Output. Pointer to where the mode of the first RNN layer should be saved.
dataType
Output. Pointer to where the data type of RNN weights/biases should be stored.
mathPrec
Output. Pointer to where the math precision type should be stored.
mathType
Output. Pointer to where the preferred option for Tensor Cores are saved.
inputSize
Output. Pointer to where the RNN input vector size is stored.
hiddenSize
Output. Pointer to where the size of the hidden state should be stored (the same value is used in every RNN layer).
projSize
Output. Pointer to where the LSTM cell output size after the recurrent projection is stored.
numLayers
Output. Pointer to where the number of RNN layers should be stored.
dropoutDesc
Output. Pointer to where the handle to a previously configured dropout descriptor should be stored.
auxFlags
Output. Pointer to miscellaneous RNN options (flags) that do not require passing additional numerical values to configure.
Returns
CUDNN_STATUS_SUCCESS
RNN parameters were successfully retrieved from the RNN descriptor.
CUDNN_STATUS_BAD_PARAM
An invalid input argument was found (rnnDesc was NULL).
CUDNN_STATUS_NOT_INITIALIZED
The cuDNN library was not initialized properly.
cudnnGetRNNTempSpaceSizes()#
This function computes the work and reserve space buffer sizes based on the RNN network geometry stored in rnnDesc, designated usage (inference or training) defined by the fMode argument, and the current RNN data dimensions (maxSeqLength, batchSize) retrieved from xDesc.  When RNN data dimensions change, the cudnnGetRNNTempSpaceSizes() must be called again because RNN temporary buffer sizes are not monotonic.
cudnnStatus_t cudnnGetRNNTempSpaceSizes(
    cudnnHandle_t handle,
    cudnnRNNDescriptor_t rnnDesc,
    cudnnForwardMode_t fMode,
    cudnnRNNDataDescriptor_t xDesc,
    size_t *workSpaceSize,
    size_t *reserveSpaceSize);
The user can assign NULL to workSpaceSize or reserveSpaceSize pointers when the corresponding value is not needed.
Parameters
handle
Input. The current cuDNN context handle.
rnnDesc
Input. A previously initialized RNN descriptor.
fMode
Input. Specifies whether temporary buffers are used in inference or training modes. The reserve-space buffer is not used during inference. Therefore, the returned size of the reserve space buffer will be zero when the fMode argument is CUDNN_FWD_MODE_INFERENCE.
xDesc
Input. A single RNN data descriptor that specifies current RNN data dimensions: maxSeqLength and batchSize




    
.
workSpaceSize
Output. Minimum amount of GPU memory in bytes needed as a workspace buffer. The workspace buffer is not used to pass intermediate results between APIs but as a temporary read/write buffer.
reserveSpaceSize
Output. Minimum amount of GPU memory in bytes needed as the reserve-space buffer. The reserve space buffer is used to pass intermediate results from cudnnRNNForward() to RNN BackwardData and BackwardWeights routines that compute first order derivatives with respect to RNN inputs or trainable weight and biases.
Returns
CUDNN_STATUS_SUCCESS
RNN temporary buffer sizes were computed successfully.
CUDNN_STATUS_BAD_PARAM
An invalid input argument was detected.
CUDNN_STATUS_NOT_SUPPORTED
An incompatible or unsupported combination of input arguments was detected.
cudnnGetRNNWeightParams()#
This function is used to obtain the start address and shape of every RNN weight matrix and bias vector in each pseudo-layer within the recurrent neural network model.
cudnnStatus_t cudnnGetRNNWeightParams(
    cudnnHandle_t handle,
    cudnnRNNDescriptor_t rnnDesc,
    int32_t pseudoLayer,
    size_t weightSpaceSize,
    const void *weightSpace,
    int32_t linLayerID,
    cudnnTensorDescriptor_t mDesc,
    void **mAddr,
    cudnnTensorDescriptor_t bDesc,
    void **bAddr);
Parameters
handle
Input. Handle to a previously created cuDNN library descriptor.
rnnDesc
Input. A previously initialized RNN descriptor.
pseudoLayer
Input. The pseudo-layer to query. In unidirectional RNNs, a pseudo-layer is the same as a physical layer (pseudoLayer=0 is the RNN input layer, pseudoLayer=1 is the first hidden layer). In bidirectional RNNs, there are twice as many pseudo-layers in comparison to physical layers:
pseudoLayer=0 refers to the forward direction sub-layer of the physical input layer
pseudoLayer=1 refers to the backward direction sub-layer of the physical input layer
pseudoLayer=2 is the forward direction sub-layer of the first hidden layer, and so on
weightSpaceSize
Input. Address of the weight space buffer. Starting from cuDNN version 9.1, this parameter can be NULL. This allows you to retrieve weight/bias offsets instead of the actual pointers within the buffer. For best performance, the recommended alignment of the weight space buffer should be 256 B or the same as returned by cudaMalloc().
weightSpace
Input. Pointer to the weight space buffer.
linLayerID
Input. Weight matrix or bias vector linear ID index.
If cellMode in rnnDesc was set to CUDNN_RNN_RELU or CUDNN_RNN_TANH:
Value 0 references the weight matrix or bias vector used in conjunction with the input from the previous layer or input to the RNN model.
Value 1 references the weight matrix or bias vector used in conjunction with the hidden state from the previous time step or the initial hidden state.
If cellMode in rnnDesc was set to CUDNN_LSTM:
Values 0, 1, 2, and 3 reference weight matrices or bias vectors used in conjunction with the input from the previous layer or input to the RNN model.
Values 4, 5, 6, and 7 reference weight matrices or bias vectors used in conjunction with the hidden state from the previous time step or the initial hidden state.
Value 8 corresponds to the projection matrix, if enabled (there is no bias in this operation).
Values and their LSTM gates:
linLayerID 0 and 4 correspond to the input gate.
linLayerID 1 and 5 correspond to the forget gate.
linLayerID 2 and 6 correspond to the new cell state calculations with hyperbolic tangent.
linLayerID 3 and 7 correspond to the output gate.
If cellMode in rnnDesc was set to CUDNN_GRU:
Values 0, 1, and 2 reference weight matrices or bias vectors used in conjunction with the input from the previous layer or input to the RNN model.
Values 3, 4, and 5 reference weight matrices or bias vectors used in conjunction with the hidden state from the previous time step or the initial hidden state.
Values and their GRU gates:
linLayerID 0 and 3 correspond to the reset gate.
linLayerID 1 and 4 reference to the update gate.
linLayerID 2 and 5 correspond to the new hidden state calculations with hyperbolic tangent.
For more information on modes and bias modes, refer to cudnnRNNMode_t.
mDesc
Output. Handle to a previously created tensor descriptor. The shape of the corresponding weight matrix is returned in this descriptor in the following format: dimA[3] = {1, rows, cols}. The reported number of tensor dimensions is zero when the weight matrix does not exist. This situation occurs for input GEMM matrices of the first layer when CUDNN_SKIP_INPUT is selected or for the LSTM projection matrix when the feature is disabled.
mAddr
Output. Pointer to the beginning of the weight matrix within the weight space buffer. When the weight matrix does not exist, the returned address written to mAddr is NULL. Starting from cuDNN version 9.1, the mDesc and mAddr arguments can be both NULL. In this case, the shape of the weight matrix and its address will not be reported. By assigning mDesc=NULL and mAddr=NULL, you can retrieve information about bias vectors only.
bDesc
Output. Handle to a previously created tensor descriptor. The shape of the corresponding bias vector is returned in this descriptor in the following format: dimA[3] = {1, rows, 1}. The reported number of tensor dimensions is zero when the bias vector does not exist.
bAddr
Output. Pointer to the beginning of the bias vector within the weight space buffer. When the bias vector does not exist, the returned address is NULL. Starting from cuDNN version 9.1, the bDesc and bAddr arguments can be both NULL. In this case, the shape of the bias vector and its address will not be reported. By assigning bDesc=NULL and bAddr=NULL, you can retrieve information about weight matrices only.
Returns
CUDNN_STATUS_SUCCESS
The query was completed successfully.
CUDNN_STATUS_BAD_PARAM
An invalid input argument was encountered.  For example, the value of pseudoLayer is out of range or linLayerID is negative or larger than 8.
CUDNN_STATUS_INVALID_VALUE
Some weight/bias elements are outside the weight space buffer boundaries.
CUDNN_STATUS_NOT_INITIALIZED
The cuDNN library was not initialized properly.
cudnnGetRNNWeightSpaceSize()#
This function reports the required size of the weight space buffer in bytes. The weight space buffer holds all RNN weight matrices and bias vectors.
cudnnStatus_t cudnnGetRNNWeightSpaceSize(
    cudnnHandle_t handle,
    cudnnRNNDescriptor_t rnnDesc,
    size_t *weightSpaceSize);
Parameters
handle
Input. The current cuDNN context handle.
rnnDesc
Input. A previously initialized RNN descriptor.
weightSpaceSize
Output. Minimum size in bytes of GPU memory needed for all RNN trainable parameters.
Returns
CUDNN_STATUS_SUCCESS
The query was successful.
CUDNN_STATUS_BAD_PARAM
An invalid input argument was encountered. For example, any input argument was NULL.
CUDNN_STATUS_NOT_INITIALIZED
The cuDNN library was not initialized properly.
cudnnGetSeqDataDescriptor()#
This function has been deprecated in cuDNN 9.0.
This function retrieves settings from a previously created sequence data descriptor. The user can assign NULL




    
 to any pointer except seqDataDesc when the retrieved value is not needed. The nbDimsRequested argument applies to both dimA[] and axes[] arrays. A positive value of nbDimsRequested or seqLengthSizeRequested is ignored when the corresponding array, dimA[], axes[], or seqLengthArray[] is NULL.
cudnnStatus_t cudnnGetSeqDataDescriptor(
    const cudnnSeqDataDescriptor_t seqDataDesc,
    cudnnDataType_t *dataType,
    int *nbDims,
    int nbDimsRequested,
    int dimA[],
    cudnnSeqDataAxis_t axes[],
    size_t *seqLengthArraySize,
    size_t seqLengthSizeRequested,
    int seqLengthArray[],
    void *paddingFill);
The cudnnGetSeqDataDescriptor() function does not report the actual strides in the sequence data buffer. Those strides can be handy in computing the offset to any sequence data element. The user must precompute strides based on the axes[] and dimA[] arrays reported by the cudnnGetSeqDataDescriptor() function. Below is sample code that performs this task:
// Array holding sequence data strides.
size_t strA[CUDNN_SEQDATA_DIM_COUNT] = {0};
// Compute strides from dimension and order arrays.
size_t stride = 1;
for (int i = nbDims - 1; i >= 0; i--) {
    int j = int(axes[i]);
    if (unsigned(j) < CUDNN_SEQDATA_DIM_COUNT-1 && strA[j] == 0) {
    strA[j] = stride;
    stride *= dimA[j];
    } else {
        fprintf(stderr, "ERROR: invalid axes[%d]=%d\n\n", i, j);
        abort();
Now, the strA[] array can be used to compute the index to any sequence data element, for example:
// Using four indices (batch, beam, time, vect) with ranges already checked.
size_t base = strA[CUDNN_SEQDATA_BATCH_DIM] * batch
            + strA[CUDNN_SEQDATA_BEAM_DIM]  * beam
        + strA[CUDNN_SEQDATA_TIME_DIM]  * time;
val = seqDataPtr[base + vect];
The above code assumes that all four indices (batch, beam, time, vect) are less than the corresponding value in the dimA[] array.  The sample code also omits the strA[CUDNN_SEQDATA_VECT_DIM] stride because its value is always 1, meaning, elements of one vector occupy a contiguous block of memory.
Parameters
seqDataDesc
Input. Sequence data descriptor.
dataType
Output. Data type used in the sequence data buffer.
nbDims
Output. The number of active dimensions in the dimA[] and axes[] arrays.
nbDimsRequested
Input. The maximum number of consecutive elements that can be written to dimA[] and axes[] arrays starting from index zero. The recommended value for this argument is CUDNN_SEQDATA_DIM_COUNT.
dimA[]
Output. Integer array holding sequence data dimensions.
axes[]
Output. Array of cudnnSeqDataAxis_t that defines the layout of sequence data in memory.
seqLengthArraySize
Output. The number of required elements in seqLengthArray[] to save all sequence lengths.
seqLengthSizeRequested
Input. The maximum number of consecutive elements that can be written to the seqLengthArray[] array starting from index zero.
seqLengthArray[]
Output. Integer array holding sequence lengths.
paddingFill
Output. Pointer to a storage location of dataType with the fill value that should be written to all padding vectors. Use NULL when an explicit initialization of output padding vectors was not requested.
Returns
CUDNN_STATUS_SUCCESS
Requested sequence data descriptor fields were retrieved successfully.
CUDNN_STATUS_BAD_PARAM
An invalid input argument was found.
CUDNN_STATUS_INTERNAL_ERROR
An inconsistent internal state was encountered.
cudnnMultiHeadAttnBackwardData()#
This function has been deprecated in cuDNN 9.0.
This function computes exact, first-order derivatives of the multihead attention block with respect to its inputs: Q, K, V.  If y=F(w) is a vector-valued function that represents the multihead attention layer and it takes some vector \(\chi\epsilon\mathbb{R}^{n}\) as an input (with all other parameters and inputs constant), and outputs vector \(\chi\epsilon\mathbb{R}^{m}\), then cudnnMultiHeadAttnBackwardData() computes the result of \(\left(\partial y_{i}/\partial x_{j}\right)^{T} \delta_{out}\) where \(\delta_{out}\) is the mx1 gradient of the loss function with respect to multihead attention outputs. The \(\delta_{out}\) gradient is back propagated through prior layers of the deep learning model. \(\partial y_{i}/\partial x_{j}\) is the mxn Jacobian matrix of F(x). The input is supplied via the dout argument and gradient results for Q, K, V are written to the dqueries, dkeys, and dvalues buffers.
The cudnnMultiHeadAttnBackwardData() function does not output partial derivatives for residual connections because this result is equal to \(\delta_{out}\). If the multihead attention model enables residual connections sourced directly from Q, then the dout tensor needs to be added to dqueries to obtain the correct result of the latter. This operation is demonstrated in the cuDNN multiHeadAttention sample code.
cudnnStatus_t cudnnMultiHeadAttnBackwardData(
    cudnnHandle_t handle,
    const cudnnAttnDescriptor_t attnDesc,
    const int loWinIdx[],
    const int hiWinIdx[],
    const int devSeqLengthsDQDO[],
    const int devSeqLengthsDKDV[],
    const cudnnSeqDataDescriptor_t doDesc,
    const void *dout,
    const cudnnSeqDataDescriptor_t dqDesc,
    void *dqueries,
    const void *queries




    
,
    const cudnnSeqDataDescriptor_t dkDesc,
    void *dkeys,
    const void *keys,
    const cudnnSeqDataDescriptor_t dvDesc,
    void *dvalues,
    const void *values,
    size_t weightSizeInBytes,
    const void *weights,
    size_t workSpaceSizeInBytes,
    void *workSpace,
    size_t reserveSpaceSizeInBytes,
    void *reserveSpace);
The cudnnMultiHeadAttnBackwardData() function must be invoked after cudnnMultiHeadAttnForward(). The loWinIdx[], hiWinIdx[], queries, keys, values, weights, and reserveSpace arguments should be the same as in the cudnnMultiHeadAttnForward() call. devSeqLengthsDQDO[] and devSeqLengthsDKDV[] device arrays should contain the same start and end attention window indices as devSeqLengthsQO[] and devSeqLengthsKV[] arrays in the forward function invocation.
cudnnMultiHeadAttnBackwardData() does not verify that sequence lengths stored in devSeqLengthsDQDO[] and devSeqLengthsDKDV[] contain the same settings as seqLengthArray[] in the corresponding sequence data descriptor.
Parameters
handle
Input. The current cuDNN context handle.
attnDesc
Input. A previously initialized attention descriptor.
loWinIdx[], hiWinIdx[]
Input. Two host integer arrays specifying the start and end indices of the attention window for each Q time-step. The start index in K, V sets is inclusive, and the end index is exclusive.
devSeqLengthsDQDO[]
Input. Device array containing a copy of the sequence length array from the dqDesc or doDesc sequence data descriptor.
devSeqLengthsDKDV[]
Input. Device array containing a copy of the sequence length array from the dkDesc or dvDesc sequence data descriptor.
doDesc
Input. Descriptor for the \(\delta_{out}\) gradients (vectors of partial derivatives of the loss function with respect to the multihead attention outputs).
dout
Input. Pointer to the \(\delta_{out}\) gradient data in the device memory.
dqDesc
Input. Descriptor for queries and dqueries sequence data.
dqueries
Output. Device pointer to gradients of the loss function computed with respect to queries vectors.
queries
Input. Pointer to queries data in the device memory. This is the same input as in cudnnMultiHeadAttnForward().
dkDesc
Input. Descriptor for keys and dkeys sequence data.
dkeys
Output. Device pointer to gradients of the loss function computed with respect to keys vectors.
keys
Input. Pointer to keys data in the device memory. This is the same input as in cudnnMultiHeadAttnForward().
dvDesc
Input. Descriptor for values and dvalues sequence data.
dvalues
Output. Device pointer to gradients of the loss function computed with respect to values vectors.
values
Input. Pointer to values data in the device memory. This is the same input as in cudnnMultiHeadAttnForward().
weightSizeInBytes
Input. Size of the weight buffer in bytes where all multihead attention trainable parameters are stored.
weights
Input. Address of the weight buffer in the device memory.
workSpaceSizeInBytes
Input. Size of the workspace buffer in bytes used for temporary API storage.
workSpace
Input/Output. Address of the workspace buffer in the device memory.
reserveSpaceSizeInBytes
Input. Size of the reserve-space buffer in bytes used for data exchange between forward and backward (gradient) API calls.
reserveSpace
Input/Output. Address to the reserve-space buffer in the device memory.
Returns
CUDNN_STATUS_SUCCESS
No errors were detected while processing API input arguments and launching GPU kernels.
CUDNN_STATUS_BAD_PARAM
An invalid or incompatible input argument was encountered.
CUDNN_STATUS_EXECUTION_FAILED
The process of launching a GPU kernel returned an error, or an earlier kernel did not complete successfully.
CUDNN_STATUS_INTERNAL_ERROR
An inconsistent internal state was encountered.
CUDNN_STATUS_NOT_SUPPORTED
A requested option or a combination of input arguments is not supported.
CUDNN_STATUS_ALLOC_FAILED
Insufficient amount of shared memory to launch a GPU kernel.
cudnnMultiHeadAttnBackwardWeights()#
This function has been deprecated in cuDNN 9.0.
This function computes exact, first-order derivatives of the multihead attention block with respect to its trainable parameters: projection weights and projection biases. If y=F(w) is a vector-valued function that represents the multihead attention layer and it takes some vector \(\chi\epsilon\mathbb{R}^{n}\) of “flatten” weights or biases as an input (with all other parameters and inputs fixed), and outputs vector \(\chi\epsilon\mathbb{R}^{m}\), then cudnnMultiHeadAttnBackwardWeights() computes the result of \(\left(\partial y_{i}/\partial w_{j}\right)^{T} \delta_{out}\) where \(\delta_{out}\) is the mx1 gradient of the loss function with respect to multihead attention outputs. The \(\delta_{out}\) gradient is back propagated through prior layers of the deep learning model. \(\partial y_{i}/\partial w_{j}\) is the mxn Jacobian matrix of F(w). The \(\delta_{out}\) input is supplied via the dout argument.
cudnnStatus_t cudnnMultiHeadAttnBackwardWeights(
    cudnnHandle_t handle,
    const cudnnAttnDescriptor_t attnDesc,
    cudnnWgradMode_t addGrad,
    const cudnnSeqDataDescriptor_t qDesc,
    const void *queries,
    const cudnnSeqDataDescriptor_t kDesc,
    const void *keys,
    const cudnnSeqDataDescriptor_t vDesc,
    const void *values,
    const cudnnSeqDataDescriptor_t doDesc,
    const void *dout,
    size_t weightSizeInBytes,
    const void *weights,
    void *dweights,
    size_t workSpaceSizeInBytes,
    void *workSpace,
    size_t reserveSpaceSizeInBytes,
    void *reserveSpace);
All gradient results with respect to weights and biases are written to the dweights buffer. The size and the organization of the dweights buffer is the same as the weights buffer that holds multihead attention weights and biases. The cuDNN multiHeadAttention sample code demonstrates how to access those weights.
Gradient of the loss function with respect to weights or biases is typically computed over multiple batches. In such a case, partial results computed for each batch should be summed together. The addGrad argument specifies if the gradients from the current batch should be added to previously computed results or the dweights




    
 buffer should be overwritten with the new results.
The cudnnMultiHeadAttnBackwardWeights() function should be invoked after cudnnMultiHeadAttnBackwardData(). The queries, keys, values, weights, and reserveSpace arguments should be the same as in cudnnMultiHeadAttnForward() and cudnnMultiHeadAttnBackwardData() calls. The dout argument should be the same as in cudnnMultiHeadAttnBackwardData().
Parameters
handle
Input. The current cuDNN context handle.
attnDesc
Input. A previously initialized attention descriptor.
addGrad
Input. Weight gradient output mode.
qDesc
Input. Descriptor for the query sequence data.
queries
Input. Pointer to queries sequence data in the device memory.
kDesc
Input. Descriptor for the keys sequence data.
keys
Input. Pointer to keys sequence data in the device memory.
vDesc
Input. Descriptor for the values sequence data.
values
Input. Pointer to values sequence data in the device memory.
doDesc
Input. Descriptor for the \(\delta_{out}\) gradients (vectors of partial derivatives of the loss function with respect to the multihead attention outputs).
dout
Input. Pointer to the \(\delta_{out}\) gradient vectors in the device memory.
weightSizeInBytes
Input. Size of the weights and dweights buffers in bytes.
weights
Input. Address of the weight buffer in the device memory.
dweights
Output. Address of the weight gradient buffer in the device memory.
workSpaceSizeInBytes
Input. Size of the workspace buffer in bytes used for temporary API storage.
workSpace
Input/Output. Address of the workspace buffer in the device memory.
reserveSpaceSizeInBytes
Input. Size of the reserve-space buffer in bytes used for data exchange between forward and backward (gradient) API calls.
reserveSpace
Input/Output. Address to the reserve-space buffer in the device memory.
Returns
CUDNN_STATUS_SUCCESS
No errors were detected while processing API input arguments and launching GPU kernels.
CUDNN_STATUS_BAD_PARAM
An invalid or incompatible input argument was encountered.
CUDNN_STATUS_EXECUTION_FAILED
The process of launching a GPU kernel returned an error, or an earlier kernel did not complete successfully.
CUDNN_STATUS_INTERNAL_ERROR
An inconsistent internal state was encountered.
CUDNN_STATUS_NOT_SUPPORTED
A requested option or a combination of input arguments is not supported.
cudnnMultiHeadAttnForward()#
This function has been deprecated in cuDNN 9.0.
The cudnnMultiHeadAttnForward() function computes the forward responses of the multihead attention layer. When reserveSpaceSizeInBytes=0 and reserveSpace=NULL, the function operates in the inference mode in which backward (gradient) functions are not invoked, otherwise, the training mode is assumed. In the training mode, the reserve space is used to pass intermediate results from cudnnMultiHeadAttnForward() to cudnnMultiHeadAttnBackwardData() and from cudnnMultiHeadAttnBackwardData() to cudnnMultiHeadAttnBackwardWeights().
cudnnStatus_t cudnnMultiHeadAttnForward(
    cudnnHandle_t handle,
    const cudnnAttnDescriptor_t attnDesc,
    int currIdx,
    const int loWinIdx[],
    const int hiWinIdx[],
    const int devSeqLengthsQO[],
    const int devSeqLengthsKV[],
    const cudnnSeqDataDescriptor_t qDesc,
    const void *queries,
    const void *residuals,
    const cudnnSeqDataDescriptor_t kDesc,
    const void *keys,
    const cudnnSeqDataDescriptor_t vDesc,
    const void *values,
    const cudnnSeqDataDescriptor_t oDesc,
    void *out,
    size_t weightSizeInBytes,
    const void *weights,
    size_t workSpaceSizeInBytes,
    void *workSpace,
    size_t reserveSpaceSizeInBytes,
    void *reserveSpace);
In the inference mode, the currIdx specifies the time-step or sequence index of the embedding vectors to be processed. In this mode, the user can perform one iteration for time-step zero (currIdx=0), then update Q, K, V vectors and the attention window, and execute the next step (currIdx=1). The iterative process can be repeated for all time-steps.
When all Q time-steps are available (for example, in the training mode or in the inference mode on the encoder side in self-attention), the user can assign a negative value to currIdx and the cudnnMultiHeadAttnForward() API will automatically sweep through all Q time-steps.
The loWinIdx[] and hiWinIdx[] host arrays specify the attention window size for each Q time-step. In a typical self-attention case, the user must include all previously visited embedding vectors but not the current or future vectors. In this situation, the user should set:
currIdx=0: loWinIdx[0]=0; hiWinIdx[0]=0;  // initial time-step, no attention window
currIdx=1: loWinIdx[1]=0; hiWinIdx[1]=1;  // attention window spans one vector
currIdx=2: loWinIdx[2]=0; hiWinIdx[2]=2;  // attention window spans two vectors
(...)
When currIdx is negative in cudnnMultiHeadAttnForward(), the loWinIdx[] and hiWinIdx[] arrays must be fully initialized for all time-steps. When cudnnMultiHeadAttnForward() is invoked with currIdx=0, currIdx=1, currIdx=2, and so on, then the user can update loWinIdx[currIdx] and hiWinIdx[currIdx] elements only before invoking the forward response function. All other elements in the loWinIdx[] and hiWinIdx[] arrays will not be accessed. Any adaptive attention window scheme can be implemented that way.
Use the following settings when the attention window should be the maximum size, for example, in cross-attention:
currIdx=0: loWinIdx[0]=0; hiWinIdx[0]=maxSeqLenK;
currIdx=1: loWinIdx[1]=0; hiWinIdx[1]=maxSeqLenK;
currIdx=2: loWinIdx[2]=0; hiWinIdx




    
[2]=maxSeqLenK;
(...)
The maxSeqLenK value above should be equal to or larger than dimA[CUDNN_SEQDATA_TIME_DIM] in the kDesc descriptor. A good choice is to use maxSeqLenK=INT_MAX from limits.h.
The actual length of any K sequence defined in seqLengthArray[] in cudnnSetSeqDataDescriptor() can be shorter than maxSeqLenK. The effective attention window span is computed based on seqLengthArray[] stored in the K sequence descriptor and indices held in loWinIdx[] and hiWinIdx[] arrays.
devSeqLengthsQO[] and devSeqLengthsKV[] are pointers to device (not host) arrays with Q, O, and K, V sequence lengths. Note that the same information is also passed in the corresponding descriptors of type cudnnSeqDataDescriptor_t on the host side. The need for extra device arrays comes from the asynchronous nature of cuDNN calls and limited size of the constant memory dedicated to GPU kernel arguments. When the cudnnMultiHeadAttnForward() API returns, the sequence length arrays stored in the descriptors can be immediately modified for the next iteration. However, the GPU kernels launched by the forward call may not have started at this point. For this reason, copies of sequence arrays are needed on the device side to be accessed directly by GPU kernels. Those copies cannot be created inside the cudnnMultiHeadAttnForward() function for very large K, V inputs without the device memory allocation and CUDA stream synchronization.
To reduce the cudnnMultiHeadAttnForward() API overhead, devSeqLengthsQO[] and devSeqLengthsKV[] device arrays are not validated to contain the same settings as seqLengthArray[] in the sequence data descriptors.
Sequence lengths in the kDesc and vDesc descriptors should be the same. Similarly, sequence lengths in the qDesc and oDesc descriptors should match. The user can define six different data layouts in the qDesc, kDesc, vDesc, and oDesc descriptors. Refer to the cudnnSetSeqDataDescriptor() function for the discussion of those layouts. All multihead attention API calls require that the same layout is used in all sequence data descriptors.
In the transformer model, the multihead attention block is tightly coupled with the layer normalization and residual connections. cudnnMultiHeadAttnForward() does not encompass the layer normalization but it can be used to handle residual connections as depicted in the following figure.
Queries and residuals share the same qDesc descriptor in cudnnMultiHeadAttnForward(). When residual connections are disabled, the residuals pointer should be NULL. When residual connections are enabled, the vector length in qDesc should match the vector length specified in the oDesc descriptor, so that a vector addition is feasible.
The queries, keys, and values pointers are not allowed to be NULL, even when K and V are the same inputs or Q, K, V are the same inputs.
Parameters
handle
Input. The current cuDNN context handle.
attnDesc
Input. A previously initialized attention descriptor.
currIdx
Input. Time-step in queries to process. When the currIdx argument is negative, all Q time-steps are processed. When currIdx is zero or positive, the forward response is computed for the selected time-step only. The latter input can be used in inference mode only, to process one time-step while updating the next attention window and Q, R, K, V inputs in-between calls.
loWinIdx[], hiWinIdx[]
Input. Two host integer arrays specifying the start and end indices of the attention window for each Q time-step. The start index in K, V sets is inclusive, and the end index is exclusive.
devSeqLengthsQO[]
Input. Device array specifying sequence lengths of query, residual, and output sequence data.
devSeqLengthsKV[]
Input. Device array specifying sequence lengths of key and value input data.
qDesc
Input. Descriptor for the query and residual sequence data.
queries
Input. Pointer to queries data in the device memory.
residuals
Input. Pointer to residual data in device memory. Set this argument to NULL if no residual connections are required.
kDesc
Input. Descriptor for the keys sequence data.
keys
Input. Pointer to keys data in the device memory.
vDesc
Input. Descriptor for the values sequence data.
values
Input. Pointer to values data in the device memory.
oDesc
Input. Descriptor for the multihead attention output sequence data.
out
Output. Pointer to device memory where the output response should be written.
weightSizeInBytes
Input. Size of the weight buffer in bytes where all multihead attention trainable parameters are stored.
weights
Input. Pointer to the weight buffer in device memory.
workSpaceSizeInBytes
Input. Size of the workspace buffer in bytes used for temporary API storage.
workSpace
Input/Output. Pointer to the workspace buffer in device memory.
reserveSpaceSizeInBytes
Input. Size of the reserve-space buffer in bytes used for data exchange between forward and backward (gradient) API calls. This parameter should be zero in the inference mode and non-zero in the training mode.
reserveSpace
Input/Output. Pointer to the reserve-space buffer in device memory. This argument should be NULL in inference mode and non-NULL in the training mode.
Returns
CUDNN_STATUS_SUCCESS
No errors were detected while processing API input arguments and launching GPU kernels.
CUDNN_STATUS_BAD_PARAM
An invalid or incompatible input argument was encountered. Some examples include:
a required input pointer was NULL
currIdx was out of bound
the descriptor value for attention, query, key, value, and output were incompatible with one another
CUDNN_STATUS_EXECUTION_FAILED
The process of launching a GPU kernel returned an error, or an earlier kernel did not complete successfully.
CUDNN_STATUS_INTERNAL_ERROR
An inconsistent internal state was encountered.
CUDNN_STATUS_NOT_SUPPORTED
A requested option or a combination of input arguments is not supported.
CUDNN_STATUS_ALLOC_FAILED
Insufficient amount of shared memory to launch a GPU kernel.
cudnnRNNBackwardData_v8()#
This function computes exact, first-order derivatives of the RNN model with respect to its inputs: x, hx and for the LSTM cell type also cx. If o = [y, hy, cy] = F(x, hx, cx) = F(z) is a vector-valued function that represents the entire RNN model and it takes vectors x (for all time-steps) and vectors hx, cx (for all layers) as inputs, concatenated into \(\textbf{z}\epsilon\mathbb{R}^{n}\) (network weights and biases are assumed constant), and outputs vectors y, hy, cy concatenated into a vector \(\textbf{o}\epsilon\mathbb{R}^{m}\), then cudnnRNNBackwardData_v8() computes the result of \(\left(\partial o_{i}/\partial z_{j}\right)^{T} \delta_{out}\) where \(\delta_{out}\) is the mx1 gradient of the loss function with respect to all RNN outputs. The \(\delta_{out}\) gradient is back propagated through prior layers of the deep learning model, starting from the model output. \(\partial o_{i}/\partial z_{j}\) is the mxn Jacobian matrix of F(z). The \(\delta_{out}\) input is supplied via the dy, dhy, and dcy arguments and gradient results \(\left(\partial o_{i}/\partial z_{j}\right)^{T} \delta_{out}\) are written to the dx, dhx, and dcx buffers.
cudnnStatus_t cudnnRNNBackwardData_v8(
    cudnnHandle_t handle,
    cudnnRNNDescriptor_t rnnDesc,
    const int32_t devSeqLengths[],
    cudnnRNNDataDescriptor_t yDesc,
    const void *y,
    const void *dy,
    




    
cudnnRNNDataDescriptor_t xDesc,
    void *dx,
    cudnnTensorDescriptor_t hDesc,
    const void *hx,
    const void *dhy,
    void *dhx,
    cudnnTensorDescriptor_t cDesc,
    const void *cx,
    const void *dcy,
    void *dcx,
    size_t weightSpaceSize,
    const void *weightSpace,
    size_t workSpaceSize,
    void *workSpace,
    size_t reserveSpaceSize,
    void *reserveSpace);
Locations of x, y, hx, cx, hy, cy, dx, dy, dhx, dcx, dhy, and dcy signals a multi-layer RNN model are shown in the following figure. Note that internal RNN signals (between time-steps and between layers) are not exposed by the cudnnRNNBackwardData_v8() function.
Memory addresses to the primary RNN output y, the initial hidden state hx, and the initial cell state cx (for LSTM only) should point to the same data as in the preceding cudnnRNNForward() call. The dy and dx pointers cannot be NULL.
The cudnnRNNBackwardData_v8() function accepts any combination of dhy, dhx, dcy, dcx buffer addresses being NULL. When dhy or dcy are NULL, it is assumed that those inputs are zero. When dhx or dcx pointers are NULL, then the corresponding results are not written by cudnnRNNBackwardData_v8(). When all hx, dhy, dhx pointers are NULL, then the corresponding tensor descriptor hDesc can be NULL too. The same rule applies to the cx, dcy, dcx pointers and the cDesc tensor descriptor.
The cudnnRNNBackwardData_v8() function allows the user to use padded layouts for inputs y, dy, and output dx. In padded or unpacked layouts (CUDNN_RNN_DATA_LAYOUT_SEQ_MAJOR_UNPACKED, CUDNN_RNN_DATA_LAYOUT_BATCH_MAJOR_UNPACKED) each sequence of vectors in a mini-batch has a fixed length defined by the maxSeqLength argument in the cudnnSetRNNDataDescriptor() function. The term “unpacked” refers here to the presence of padding vectors, and not unused address ranges between contiguous vectors.
Each padded, fixed-length sequence starts from a segment of valid vectors. The valid vector count is stored in seqLengthArray passed to cudnnSetRNNDataDescriptor(), such that 0 < seqLengthArray[i] <= maxSeqLength for all sequences in a mini-batch, that is, for i=0..batchSize-1. The remaining, padding vectors make the combined sequence length equal to maxSeqLength. Both sequence-major and batch-major padded layouts are supported. In addition, a packed sequence-major layout is supported: CUDNN_RNN_DATA_LAYOUT_SEQ_MAJOR_PACKED.
In the latter layout, sequences of vectors in a mini-batch are sorted in the descending order according to the sequence lengths. First all vectors for time step zero are stored. They are followed by all vectors for time step one, and so on. This layout uses no padding vectors.
The same layout type must be specified in xDesc and yDesc descriptors.
Two host arrays named seqLengthArray in xDesc and yDesc RNN data descriptors must be the same. In addition, a copy of seqLengthArray in the device memory must be passed via the devSeqLengths argument. This array is supplied directly to GPU kernels. Starting in cuDNN 8.9.1, the devSeqLengths parameter is no longer required and can be set to NULL. The variable sequence length array is transferred automatically to GPU memory by the cudnnRNNBackwardData_v8() function.
The cudnnRNNBackwardData_v8() function does not verify that sequence lengths stored in devSeqLengths in GPU memory are the same as in xDesc and yDesc descriptors in CPU memory. Sequence length arrays from xDesc and yDesc descriptors are checked for consistency, however.
The cudnnRNNBackwardData_v8() function must be called after cudnnRNNForward(). The cudnnRNNForward() function should be invoked with the fwdMode argument of type cudnnForwardMode_t set to CUDNN_FWD_MODE_TRAINING.
Parameters
handle
Input. The current cuDNN context handle.
rnnDesc
Input. A previously initialized RNN descriptor.
devSeqLengths
Input. A copy of seqLengthArray from xDesc or yDesc RNN data descriptors. The devSeqLengths array must be stored in GPU memory as it is accessed asynchronously by GPU kernels, possibly after the cudnnRNNBackwardData_v8() function exists. In cuDNN 8.9.1 and later versions, devSeqLengths should be NULL.
yDesc
Input. A previously initialized descriptor corresponding to the RNN model primary output. The dataType, layout, maxSeqLength, batchSize, and seqLengthArray need to match that of xDesc.
y, dy
Input. Data pointers to GPU buffers holding the RNN model primary output and gradient deltas (gradient of the loss function with respect to y). The y output should be produced by the preceding cudnnRNNForward() call. The y and dy vectors are expected to be laid out in memory according to the layout specified by yDesc. The elements in the tensor (including elements in padding vectors) must be densely packed. The y and dy arguments cannot be NULL.
xDesc
Input. A previously initialized RNN data descriptor corresponding to the gradient of the loss function with respect to the RNN primary model input. The dataType, layout, maxSeqLength, batchSize, and seqLengthArray must match that of yDesc. The parameter vectorSize must match the inputSize argument passed to the cudnnSetRNNDescriptor_v8() function.
dx
Output. Data pointer to GPU memory where back-propagated gradients of the loss function with respect to the RNN primary input x should be stored. The vectors are expected to be arranged in memory according to the layout specified by xDesc. The elements in the tensor (including padding vectors) must be densely packed. This argument cannot be NULL.
hDesc
Input. A tensor descriptor describing the initial RNN hidden state hx and gradient deltas dhy, dhx of the loss function. Hidden state data and gradients must be fully packed. The first dimension of the tensor depends on the dirMode argument passed to the cudnnSetRNNDescriptor_v8() function.
If dirMode is CUDNN_UNIDIRECTIONAL, then the first dimension should match the numLayers argument passed to cudnnSetRNNDescriptor_v8().
If dirMode is CUDNN_BIDIRECTIONAL, then the first dimension should be double the numLayers argument passed to cudnnSetRNNDescriptor_v8().
The second dimension must match the batchSize parameter described in xDesc. The third dimension depends on whether RNN mode is CUDNN_LSTM and whether the LSTM projection is enabled. Specifically:
If RNN mode is CUDNN_LSTM and LSTM projection is enabled, the third dimension must match the projSize argument.
Otherwise, the third dimension must match the hiddenSize argument.
hx, dhy
Input. Addresses of GPU buffers with the RNN initial hidden state hx and gradient deltas dhy. Data dimensions are described by the hDesc tensor descriptor. If a NULL pointer is passed in hx or dhy arguments, the corresponding buffer is assumed to contain all zeros.
dhx
Output. Pointer to the GPU buffer where first-order derivatives corresponding to initial hidden state variables should be stored. Data dimensions are described by the hDesc tensor descriptor. If a NULL pointer is assigned to dhx, the back-propagated derivatives are not saved.
cDesc
Input. For LSTM networks only. This argument should be NULL for RELU, TANH, or GRU cell types. cDesc is a tensor descriptor specifying buffer layouts of the initial cell state cx and gradient deltas dcy, dcx of the loss function. Cell state data must be fully packed. The first dimension of the tensor depends on the dirMode argument passed to the cudnnSetRNNDescriptor_v8() call.
If dirMode is CUDNN_UNIDIRECTIONAL, then the first dimension should match the numLayers argument passed to cudnnSetRNNDescriptor_v8().
If dirMode is CUDNN_BIDIRECTIONAL, then the first dimension should be double the numLayers argument passed to cudnnSetRNNDescriptor_v8().
The second tensor dimension must match the batchSize parameter in xDesc. The third dimension must match the hiddenSize argument passed to the cudnnSetRNNDescriptor_v8() call.
cx, dcy
Input. For LSTM networks only. Addresses of GPU buffers with the initial LSTM state data and gradient deltas dcy. Data dimensions are described by the cDesc tensor descriptor. If a NULL pointer is passed in cx or dcy arguments, the corresponding buffer is assumed to contain all zeros.
dcx
Output. For LSTM networks only. Pointer to the GPU buffer where first-order derivatives corresponding to initial LSTM state variables should be stored. Data dimensions are described by the cDesc tensor descriptor. If a NULL pointer is assigned to dcx, the back-propagated derivatives are not saved.
weightSpaceSize
Input. Specifies the size in bytes of the provided weight space buffer.
weightSpace
Input. Address of the weight space buffer in GPU memory.
workSpaceSize
Input. Specifies the size in bytes of the provided workspace buffer.
workSpace
Input/Output. Address of the workspace buffer in GPU memory to store temporary data.
reserveSpaceSize
Input. Specifies the size in bytes of the reserve-space buffer.
reserveSpace
Input/Output. Address of the reserve-space buffer in GPU memory.
Returns
CUDNN_STATUS_SUCCESS
No errors were detected while processing API input arguments and launching GPU kernels.
CUDNN_STATUS_NOT_SUPPORTED
At least one of the following conditions are met:
variable sequence length input is passed while CUDNN_RNN_ALGO_PERSIST_STATIC, CUDNN_RNN_ALGO_PERSIST_DYNAMIC, or CUDNN_RNN_ALGO_PERSIST_STATIC_SMALL_H is specified
CUDNN_RNN_ALGO_PERSIST_STATIC or CUDNN_RNN_ALGO_PERSIST_DYNAMIC is requested on pre-Pascal devices
the ‘double’ floating point type is used for input/output and the CUDNN_RNN_ALGO_PERSIST_STATIC algo
CUDNN_STATUS_BAD_PARAM
An invalid or incompatible input argument was encountered. Some examples include:
some descriptors or data buffer addresses are NULL
settings in rnnDesc, xDesc, yDesc, hDesc, or cDesc descriptors are invalid
weightSpaceSize, workSpaceSize, or reserveSpaceSize is too small
CUDNN_STATUS_MAPPING_ERROR
A GPU/CUDA resource, such as a texture object, shared memory, or zero-copy memory is not available in the required size or there is a mismatch between the user resource and cuDNN internal resources. A resource mismatch may occur, for example, when calling cudnnSetStream(). There could be a mismatch between the user provided CUDA stream and the internal CUDA events instantiated in the cuDNN handle when cudnnCreate() was invoked.
This error status may not be correctable when it is related to texture dimensions, shared memory size, or zero-copy memory availability. If CUDNN_STATUS_MAPPING_ERROR is returned by cudnnSetStream(), then it is typically correctable, however, it means that the cuDNN handle was created on one GPU and the user stream passed to this function is associated with another GPU.
CUDNN_STATUS_EXECUTION_FAILED
The process of launching a GPU kernel returned an error, or an earlier kernel did not complete successfully.
CUDNN_STATUS_ALLOC_FAILED
The function was unable to allocate CPU memory.
cudnnRNNBackwardWeights_v8()#
This function computes exact, first-order derivatives of the RNN model with respect to all trainable parameters: weights and biases.  If o = [y, hy, cy] = F(w) is a vector-valued function that represents the multi-layer RNN model and it takes some vector \(\textbf{w}\epsilon\mathbb{R}^{n}\) of “flatten” weights or biases as input (with all other data inputs constant), and outputs vector \(\textbf{o}\epsilon\mathbb{R}^{m}\), then cudnnRNNBackwardWeights_v8() computes the result of \(\left(\partial o_{i}/\partial w_{j}\right)^{T} \delta_{out}\) where \(\delta_{out}\) is the mx1 gradient of the loss function with respect to all RNN outputs. The \(\delta_{out}\) gradient is back propagated through prior layers of the deep learning model, starting from the model output. \(\partial o_{i}/\partial w_{j}\) is the mxn Jacobian matrix of F(w). The \(\delta_{out}\) input is supplied via the dy, dhy, and dcy arguments in the cudnnRNNBackwardData_v8() function.