一身肌肉的钥匙扣 · Python对文本进行分句_python 分句· 1 月前 · |
一身肌肉的钥匙扣 · EF MySQL未将引用设置到对象的实例 ...· 9 月前 · |
一身肌肉的钥匙扣 · Data governance ...· 9 月前 · |
一身肌肉的钥匙扣 · javascript - In ...· 9 月前 · |
一身肌肉的钥匙扣 · html用代码制作虚线框怎么做? ...· 9 月前 · |
一身肌肉的钥匙扣 · 道明银行(TD)股票股价,行情,新闻,财报, ...· 10 月前 · |
一身肌肉的钥匙扣 · BIG-IP 15.1.2 Fixes ...· 10 月前 · |
一身肌肉的钥匙扣 · IPS Web服務WSDL版本 | ...· 10 月前 · |
一身肌肉的钥匙扣 · 如何计算 SASS 中的百分比减 px? 开发文档· 11 月前 · |
一身肌肉的钥匙扣 · SNMP Config File ...· 11 月前 · |
安静的李子 · 调用ModifyDBInstanceConn ...· 3 分钟前 · |
性感的佛珠 · System.ArgumentExcepti ...· 3 分钟前 · |
大力的镜子 · 不支持关键字: “connection ...· 3 分钟前 · |
痴情的可乐 · C#实体框架:关键字不支持:‘端口’开发者社区· 3 分钟前 · |
文质彬彬的凉面 · Fragment懒加载还在用setUserV ...· 3 分钟前 · |
热心肠的鸡蛋 · 【已解决】ORA-00942: table ...· 1小时前 · |
暗恋学妹的投影仪 · 《spring ...· 1小时前 · |
无邪的铁板烧 · mybatis 操作 oracle 报错 ...· 1小时前 · |
暗恋学妹的长颈鹿 · 如何解决用ESLint验证导致报错的问题_f ...· 1小时前 · |
Component: TMOS
Symptoms:
TMSH Command: "list mgmt shared <tab>" does not display the tab complete option. You may see an error:
(tmos)# list mgmt shared echo *tab*
Unexpected Error: "Object contains no "method" child value"
Conditions:
When mgmt is used in a tmsh command and you attempt to tab complete
Impact:
You are unable to configure objects in mgmt.
This issue also prevents users with the admin role from accessing the following REST endpoints:
shared/authz/users
shared/echo-js
The error returned was HTTP/1.1 401 F5 Authorization Required
Component: Local Traffic Manager
Symptoms:
- A bursty retransmission occurs during TCP's loss recovery period.
Conditions:
- TCP filter is used.
- TCP stack is used instead of TCP4 stack (based on profile settings).
- Packet loss occurs during the data transfer and TCP's loss recovery takes place.
Impact:
The bursty retransmissions may lead to more data getting lost due to large amount of data being injected into the network.
Workaround:
In versions prior to v16.0.0, use a TCP profile which selects the TCP4 stack instead of the TCP stack. There is no workaround for version 16.0.0.
Component: Local Traffic Manager
Symptoms:
A snat-translation address with ARP enabled responds to ARP requests but not ICMP ECHO requests.
Conditions:
A snat-translation address is configured with ARP enabled.
Impact:
Application traffic should not be impacted, as external hosts trying to resolve the snat-translation and return traffic to it should still be able to do so; however, ping is an important network troubleshooting tool, and not being able to ping the address may create confusion.
Workaround:
None.
Component: Anomaly Detection Services
Symptoms:
Admd consumes more than 10GB of RSS
Wrong signature statistics and possible memory corruption, potentially results in high memory consumption.
Conditions:
-- Upgrading from 13.x, 14.x to 15.x
-- Device service clustering configuration
-- App-Protect-DOS signatures configured.
Impact:
ADMD not working, ADMD constantly restarting, consuming all of the system memory. Out of memory. ADMD killed due to memory consumption
Workaround:
Make sure that all the devices within a cluster are running compatible state file version (either all with versions before 15.1.0.x or after), if not, then:
1. Stop ADMD on all of those devices: bigstart stop admd
2. Upgrade or Downgrade the BIG-IP version to match the above criteria.
3. Remove the old state files: rm -rf /var/run/adm/*
4. Start ADMD: bigstart start ADMD
If there is an issue on a single blade device, then:
1. Stop ADMD on all of those machines: bigstart stop admd
2. Remove the old state files: rm -rf /var/run/adm/*
3. Start ADMD: bigstart start ADMD
Component: TMOS
Symptoms:
In a rare scenario, a B4450 blade suddenly reboots.
Conditions:
This occurs when there is heavy traffic load on VIPRION B4450 blades. The root cause is unknown. It happens extremely rarely.
Impact:
Traffic disrupted while the blade reboots.
Workaround:
None.
Component: Application Security Manager
Symptoms:
Any web-socket request with JSON payload may cause a core witihin the JSON parser, depending on the used machine memory distribution.
Conditions:
Depends on the memory distribution of the used machine.
Sending web-socket request with JSON payload to the backend server.
Impact:
BD crash while parsing the JSON payload.
Workaround:
No crashes during JSON payload parsing.
Component: Advanced Firewall Manager
Symptoms:
Time stamp cookies and hardware accelerated flows are mutually exclusive.
Conditions:
Time stamp cookie enabled for TCP flows on a VLAN with hardware offload enabled as well.
Impact:
Reduced traffic throughput and increased CPU usage
Component: Application Security Manager
Symptoms:
When multiple Logging Profiles profile changes are made in a single update, the previous Logging Profiles are not disassociated from the virtual server. Additionally, when an Application Security Logging Profile change is made, newly added Protocol Security Logging Profile settings do not take effect.
Conditions:
Multiple Logging Profile changes are made in a single update.
Impact:
The previous Logging Profiles are not disassociated from the virtual server.
Workaround:
Perform each Log Profile change individually. For example, to change an Application Security Log Profile:
1. Remove the current association and save.
2. Add the new association and save again.
Component: Local Traffic Manager
Symptoms:
In some cases, TMM might crash when processing MPTCP traffic.
Conditions:
A TCP profile with 'Multipath TCP' enabled is attached to a virtual server.
Impact:
Traffic interrupted while TMM restarts.
Workaround:
There is no workaround other than to disable MPTCP.
Component: Application Security Manager
Symptoms:
False positive violations are detected for incorrect parameter in case of "Maximum Array/Object Elements" is reached with enabled "Parse Parameter".
Conditions:
"JSON data does not comply with format settings" and "Illegal meta character in value" violations are enabled and content profile parsing is enabled in ASM.
Impact:
False positives detected, such as "Illegal meta character in value" violation and attack signature for incorrect context.
Workaround:
No false positives detected.
Component: Fraud Protection Services
Symptoms:
MobileSafe SDK does not support iOS jailbreak detection, so the GUI should refer only to Android Rooting Detection.
Conditions:
-- Fraud Protection Service (FPS) provisioned.
-- FPS and MobileSafe Licensed.
Impact:
Introduces confusion when indicating that iOS jailbreak detection is supported, which it is not.
Workaround:
None.
Component: Application Security Manager
Symptoms:
If "Maximum Array/Object Elements" is reached and "JSON data does not comply with format settings" is detected, then all sensitive
data after last allowed element is not masked.
Conditions:
Define JSON profile, set "JSON data does not comply with format settings" to blocking and set "Maximum Array/Object Elements" to desired value.
Impact:
Data after last allowed element is not masked.
Component: Local Traffic Manager
Symptoms:
When an HTTP/2 profile is configured on the client side, chunked responses are always sent unchunked. When a connection to a client is congested, the BIG-IP system may not properly close established server-side connections causing subsequent HTTP/2 requests to stall.
Conditions:
-- A virtual server with an HTTP/2 profile configured on the client side.
-- A server responds with a chunked response.
Impact:
HTTP/2 requests intermittently stall due to the existing server-side TCP connection remaining open.
Workaround:
Configure an HTTP profile on the client side with a value of 'unchunk' on the response-chunking option.
Component: Local Traffic Manager
Symptoms:
After HTTP::respond iRule, when "Connection: close" header is sent to the client, TCP connection is not closed.
Conditions:
- TCP profile is used.
- HTTP profile is used.
- HTTP::respond iRule is used (via HTTP_RESPONSE).
- HTTP sends "Connection: close" header.
Impact:
TCP connection lives longer than needed.
Workaround:
TCP connection is closed when necessary after responding with HTTP::respond iRule.
Component: Advanced Firewall Manager
Symptoms:
TMM crashes while running traffic that uses AFM Firewall policies.
Conditions:
-- Update IP geolocation database file to the latest version.
-- Configure AFM policies with logging enabled.
-- Run traffic which hits the AFM policies and triggers logging.
Impact:
Traffic disrupted while tmm restarts.
Workaround:
Revert to using the previously working version of the IP-geolocation file.
https://support.f5.com/csp/article/K11176#restore
Component: Advanced Firewall Manager
Symptoms:
Port Block Update log message is missing the "Start time" field.
Conditions:
-- Configure PBA mode in AFMNAT/CGNAT with subscriber awareness.
-- Trigger PBA Update log messages with change in susbsriber name for the same client IP address.
Impact:
NAT Log information is not usable for accounting purpose.
Component: TMOS
Symptoms:
In rare cases where the HSB detects an error and triggers an high availability (HA) failover, HSB-specific diagnostic data is not provided.
An example are XLMAC errors, which can be seen in the LTM logs:
<13> Jul 25 18:49:41 notice The number of the HSB XLMAC recovery operation 11 or fcs failover count 0 reached threshold 11 on bus: 3.
<13> Jul 25 18:49:41 notice high availability (HA) failover action is triggered due to XLMAC/FCS erros on HSB1 on bus 3.
Conditions:
The HSB detects an internal error.
Impact:
There is less HSB data for analysis when an internal HSB occurs.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
FastL4 profiles configured with the TCP timestamp rewrite option enabled does not treat retransmitted SYNs in a correct manner.
Conditions:
FastL4 profile with TCP timestamp rewrite option is in use.
Impact:
Timestamp on some TCP packets sent by BIG-IP systems might be incorrect.
Workaround:
Do not use TCP timestamp rewrite.
Component: Carrier-Grade NAT
Symptoms:
Under certain conditions, TMM may crash while processing IPv6 NAT traffic
Conditions:
-IPv6 NAT configured
Impact:
TMM crash leading to a failover event.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
The HTTP/2 protocol allows informing a peer about the number of concurrent streams it is allowed to have. When this number is exceeded, the RFC stipulates that the system must serve all open streams and then terminate a connection.
Conditions:
-- The BIG-IP system has a virtual server with an HTTP/2 profile configured on the client side.
-- A client opens more streams than a configured value for concurrent-streams-per-connection in HTTP/2 profile.
Impact:
BIG-IP resets a connection and a client (browser) does not receive any response for outstanding requests. It requires manually reload of the webpage to address the issue.
Workaround:
None.
Component: TMOS
Symptoms:
TMM cores.
Conditions:
FastL4's hardware offloading is used.
Because the error is an internal software logic implementation, there is no direct specific configuration that triggers this error condition. A quick traffic spike during a short period of time makes it more likely to occur.
Impact:
TMM cores and the system cannot process traffic. Traffic disrupted while tmm restarts.
Workaround:
Disable PVA/EPVA on all FastL4 profiles
Component: Fraud Protection Services
Symptoms:
FPS GUI does not follow current best practices.
Conditions:
* Provision and license Fraud Protection Service (FPS).
Impact:
FPS GUI does not follow current best practices.
Workaround:
None.
Component: TMOS
Symptoms:
ZebOS BGP might not properly clear the multi-path candidate flag when handling a BGP route.
Conditions:
An inbound route-map exists that modifies a route's path selection attribute.
Impact:
Incorrect path selection and/or a timer on a route getting refreshed every time the Routing Information Base (RIB) is scanned.
Workaround:
None.
Component: TMOS
Symptoms:
After applying an Engineering Hotfix ISO that contains the tomcat package, live-update files are inadvertently removed and live update no longer works properly.
Conditions:
-- Engineering Hotfix contains the tomcat package.
-- Engineering Hotfix installation.
Impact:
Live-update functionality does not work properly.
Workaround:
There is no workaround to avoid this issue. But you can reinstall a newly created Engineering Hotfix that uses a fixed version of the live-install package.
Component: Local Traffic Manager
Symptoms:
After an HTTP return code of 400 Bad Request or 403 Forbidden, connection table entries may not be removed from the connection table until they reach the idle timeout threshold.
Conditions:
-- HTTP Explicit Proxy Configuration.
-- BIG-IP HTTP response contains a 400 Bad Request or 403 Forbidden status code.
Impact:
The hanging connection table entries can cause subsequent connections from the same source port to fail. Also, the subsequent connection attempts can cause the idle timer to be reset.
Workaround:
Use an iRule to prevent connections from hanging:
when HTTP_REJECT {
after 1
HTTP Explicit Proxy configurations no longer results in connections hanging until idle timeout.
Component: TMOS
Symptoms:
The '@' character is no longer valid in SNMP community strings.
Conditions:
Attempting to use the '@' character in SNMP community strings.
Impact:
Unable to use the '@' character in SNMP community strings. The system cannot process SNMP commands with community strings that contain the '@' character, and the commands fail.
Workaround:
Use a community string that does not contain the '@' character.
Component: Device Management
Symptoms:
iControl REST framework does not follow best practices for exception handling
Conditions:
iControl REST framework use by an authenticated user.
Impact:
iControl REST framework does not follow best practices for exception handling
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
When an HTTP/2 profile is configured on the client side, chunked responses are always sent unchunked. When a connection to a client is congested, BIG-IP systems may send the END_STREAM flag before transmitting a whole payload.
Conditions:
-- A virtual server with an HTTP/2 profile configured on the client side.
-- A server responds with a chunked response.
Impact:
A browser may not receive the whole payload, or it may not recognize that the payload has been delivered fully (partially prior to the DATA frame with END_STREAM flag, partially after the frame).
Workaround:
Configure an HTTP profile on the client side with a value of 'unchunk' on the response-chunking option.
Component: Local Traffic Manager
Symptoms:
Under certain conditions, TMM may crash while processing HTTP traffic.
Conditions:
A virtual with standard HTTP profile is configured.
Impact:
When TMM crashes it causes a failover event and may interrupt traffic processing.
Workaround:
TMM now processes HTTP traffic as expected.
Component: TMOS
Symptoms:
If there is a truncated BIG-IP software image in /shared/images, a kernel lockup and reboot could occur.
One way to have a truncated image in /shared/images is by using iControl/SOAP to upload the image. Using SOAP, the image is uploaded in chunks, so until the last chunk is uploaded, the image is not complete/is truncated.
Conditions:
-- Truncated BIG-IP image in /shared/images
-- Using SOAP to upload the image.
Impact:
Traffic disruption caused by the reboot.
Workaround:
If you are using SOAP to upload BIG-IP software images, upload them to /shared first and then move them to /shared/images.
Component: TMOS
Symptoms:
Under certain conditions, the Hardware Security Module UI does not follow current best practices.
Conditions:
Authenticated access of the HSM UI
Impact:
Under certain conditions, the Hardware Security Module UI does not follow current best practices.
Workaround:
None.
Component: Access Policy Manager
Symptoms:
Under certain conditions APM does not follow current best practices.
Conditions:
Undisclosed conditions.
Impact:
Under certain conditions APM does not follow current best practices.
Workaround:
None.
Component: TMOS
Symptoms:
1. After BIG-IP reboot, vxnet interfaces come up as uninitialized.
2. The driver does not log any issues:
echo "device driver [client-specific driver info] mlxvf5" >> /config/tmm_init.tcl
Conditions:
Running BIG-IP Virtual Edition (VE) v15.1.0.4 software.
Impact:
Vxnet driver requires manual intervention after reboot.
Workaround:
Tmsh enable/disable interface brings it back up until next reboot.
Component: Application Security Manager
Symptoms:
Request that should be passed to the backend server with cookie header which contain cookie valid value encoded to base64 is blocked.
Conditions:
A cookie name has to be defined in "Security ›› Application Security : Headers : Cookies List ›› New Cookie..." with enabled "Base64 Decoding".
Impact:
Blocking page, while the request should not be blocked.
Workaround:
Disable "Base64 Decoding" for the desired cookie.
Component: TMOS
Symptoms:
Installation fails with a 'Disk full (volume group)' error in var/log/liveinstall.log:
error: tm_install::Process::Process_full_install -- predicted size for BIGIP14125 is 12315728, current location size is 11120640, and vg has 0 remaining.
Conditions:
Platforms with software RAID that also have a symlink in /dev/md that looks like the following:
[root@bigip1] images # ls -l /dev/md/
total 8
-rw-r--r--. 1 root root 5 2020-07-09 16:12 autorebuild.pid
lrwxrwxrwx. 1 root root 8 2020-07-09 16:51 localhost:0 -> ../md127
-rw-------. 1 root root 66 2020-07-09 16:11 md-device-map
Impact:
Unable to successfully upgrade.
Workaround:
Create the expected symlink manually:
cd /dev/md
ln -s ../md127 _none_\:0
Component: Local Traffic Manager
Symptoms:
QUIC HANDSHAKE_DONE profile statistics are not set back to 0 when statistics are reset.
Conditions:
A QUIC virtual server receives or sends HANDSHAKE_DONE frames, and the profile statistics are later reset.
Impact:
QUIC HANDSHAKE_DONE profile statistics are not reset.
Workaround:
Restart tmm to reset all statistics:
Impact of Workaround: Traffic disrupted while tmm restarts.
bigstart restart tmm
Component: Global Traffic Manager (DNS)
Symptoms:
The GTM/DNS gateway_icmp monitor for IPv6 virtual servers sometimes returns 'state: timeout' even though big3d receives the packet successfully.
Conditions:
- GTM/DNS provisioned.
- IPv6 virtual server with gateway_icmp GTM/DNS monitor.
Impact:
IPv6 virtual servers are marked down unexpectedly.
Workaround:
Use a different gtm monitor type than gateway_icmp for IPv6 targets
Component: Local Traffic Manager
Symptoms:
After upgrade to v15.1.0.4, config does not load. Logs show:
-- err mcpd[11863]: 01b50049:3: FipsUserMgr Error: Master key load failure.
-- err mcpd[11863]: 01070712:3: Caught configuration exception (0), FIPS 140 operations not available on this system.
-- err tmsh[14528]: 01420006:3: Loading configuration process failed.
Conditions:
-- Upgrading to v15.1.0.4.
-- Using the following platforms:
+ i5820-DF / i7820-DF
+ 5250v-F / 7200v-F
+ 10200v-F
+ 10350v-F
Impact:
Cannot upgrade to v15.1.0.4, and the system is offline.
Important: Although you cannot prevent this from happening (except by not upgrading to 15.1.0.4), you can boot back into the previous configuration to recover BIG-IP system operation.
Workaround:
None.
Component: Access Policy Manager
Symptoms:
TCP connection reset when 'Logout URI Include' configured.
Conditions:
-- Access Policy with a valid 'Logout URI Include' string, e.g.:
/logoff.html
-- Request to 'Logout URI Include' URI from user-agent that includes a query parameter string, e.g.:
/logoff.html?a=b
Impact:
TCP connection resets, reporting BIG-IP APM error messages.
'Logout URI Include' does not support custom query strings in logout URIs to include. For example, with a 'Logout URI Include' value of /logoff.html, if a user-agent sends a logout URI request in the form of /logoff.html?a=b, logout URI validation resets the connection and reports an error:
-- Access encountered error: ERR_ARG. File: ../modules/hudfilter/access/access.c, Function: access_check_uri_type.
Note: BIG-IP APM prohibits the configuration of 'Logout URI Include' from containing a query string on the BIG-IP system. For example, attempting to configure 'Logout URI Include' with a URI in the form of /logoff.html?a=b fails and displays error messages:
-- Configuration error: Configured URI (/logoff.html?a=b) is not allowed to contain query parameter.
Workaround:
The system now ignores unsupported query parameters.
Component: TMOS
Symptoms:
The End User License Agreement (EULA) presented in VMware is out of date.
Conditions:
The EULA is presented to the user when deploying an OVF template.
Impact:
The current EULA is version: DOC-0355-16 (as explained in K12902: End User License Agreement Change Notice :: https://support.f5.com/csp/article/K12902).
Although the OVA EULA for 16.0.0 shows: DOC-0355-12, the EULA presented during license activation is the EULA in force for this instance, so you can safely ignore the discrepancy; there is no functional impact.
Workaround:
None needed. The EULA presented during license activation is the EULA in force for this instance.
Component: TMOS
Symptoms:
While restoring a UCS archive, you get an error similar to the following example:
/var: Not enough free space
535162880 bytes required
326418432 bytes available
/shared/my.ucs: Not enough free disk space to install!
Operation aborted.
/var/tmp/configsync.spec: Error installing package
Config install aborted.
Unexpected Error: UCS loading process failed.
As part of restoring UCS archives, some files (for example, the contents of the filestore) are temporarily copied to the /var/tmp directory.
The script that ensures enough free disk space is available for the UCS restore operation incorrectly reports the /var filesystem's free disk space for the /var/tmp directory.
This is incorrect, as /var/tmp is a symlink to /shared/tmp, and so the free disk space of the /shared filesystem should be used instead.
Conditions:
-- Restoring a UCS file.
-- The UCS file contains large items that are temporarily stored under the /var/tmp directory (for example, many EPSEC files, many large external data-groups, etc.).
-- The /var filesystem has limited free disk space.
Impact:
The UCS installation fails even if /var/tmp has sufficient disk space.
Workaround:
None.
Component: Service Provider
Symptoms:
Current Diameter CER/CEA messages does not advertise all HostIPAddresses.
Conditions:
-- Exchange Diameter messages CER/CEA between peers, configure a SNAT pool and an alternate address in the SCTP profile.
-- The CER from BIG-IP contains snatpool IP addresses
-- The CEA from BIG-IP contains alternate addresses
Impact:
Unable to see multiple HostIPAddress in CER/CEA
Component: Application Visibility and Reporting
Symptoms:
Some of the calculations are inaccurate/missing in the AVR publisher for DNS and SIP, and incorrect values are shown in the REST response.
Conditions:
-- Device vector detection and mitigation thresholds are set to 10.
-- A detection and mitigation threshold is reached
Impact:
An incorrect value is calculated in the REST response.
Component: Anomaly Detection Services
Symptoms:
Top shows that a large number of admd processes are running.
Conditions:
-- Configuration with Sync-Failover device groups and BADOS.
-- Some stressful (unknown) condition occurs.
Impact:
Memory is exhausted.
Workaround:
Restart admd:
bigstart restart admd
Component: Local Traffic Manager
Symptoms:
UDP profiles have a setting to set the Maximum Buffer Packets for UDP connections. This value cannot be modified with an iRule.
Conditions:
-- UDP profile is used.
-- You need to dynamically change the max buffer packets setting in an iRule.
Impact:
Unable to dynamically change the max buffer packets setting in an iRule.
Workaround:
You can now dynamically change the max buffer packets setting in an iRule. The setting is UDP::max_buf_pkts
Behavior Change:
A new iRule command has been added, UDP::max_buf_pkts. This allows you to dynamically override the maximum number of packets setting in the UDP profile.
Component: TMOS
Symptoms:
Rotated log files that are modified immediately after log rotation and before signature generation can cause signature verification failure.
Conditions:
-- Log integrity feature is enabled.
-- A log rotation event occurs
Impact:
Signature verification may fail on rotated log files.
Component: TMOS
Symptoms:
Unable to see the full name of the SSL client and SSL server profiles when assigning them in the GUI.
Conditions:
In Local Traffic :: Virtual Server :: Properties, the fields for the 'Selected' and 'Available' lists are narrower than they were in previous versions.
Impact:
With longer SSL profile names, the full name is not visible. Even the default, provided profiles, such as crypto-server-default-clientssl and crypto-client-default-serverssl, are truncated.
Note: The fields remain at the limited width even when the browser window is maximized.
Workaround:
Use tmsh to see the full SSL client and SSL server name.
Component: Application Security Manager
Symptoms:
ASM processes some web-socket requests longer than usual and can cause some latency when passing requests to the backend server.
Conditions:
Content profile parsing is enabled in ASM.
Impact:
Latency can be noted when passing web-socket requests to the backend server.
Workaround:
ASM is processes web-socket requests as expected
Component: Application Security Manager
Symptoms:
The security policies assigned to a virtual server are different among the devices in a traffic-group.
Conditions:
-- ASM provisioned.
-- Manual Sync Active-Standby Failover Device Group with ASM sync enabled.
-- An L7 ASM security policy is manually changed on a virtual server (not using the ASM wizard).
Impact:
After incremental sync, devices report 'In Sync' but there is a configuration discrepancy in the security policy assigned to the virtual server.
Workaround:
Modify the underlying LTM policy to be 'legacy':
# tmsh modify ltm policy <LTM Policy Name> legacy
Behavior Change:
There is now an internal config parameter that enables a workaround for this issue. In order to use the workaround, you must enable a db variable.
To enable the workaround, run the following command from the CLI on every device in the device group:
------------------------------------
# /usr/share/ts/bin/add_del_internal add force_legacy_ltm_policy 1
Operation completed successfully. Don't forget to restart ASM to apply changes.
------------------------------------
NOTE: In this specific case, ASM restart is not required, despite the fact that a message says it is.
Component: Advanced Firewall Manager
Symptoms:
'Traffic Statistics' syslog/Splunk messages are sent with the hostname of the standby device.
Conditions:
When a virtual server is configured with a security logging profile enabled for DoS Protection logging.
Impact:
'Traffic Statistics' syslog/Splunk messages show the wrong hostname. It should show the active device hostname.
Workaround:
None.
Component: TMOS
Symptoms:
When the device has high CPU or I/O rate, it can cause the JavaScript Obfuscator to run multiple times simultaneously, causing even higher CPU usage.
Conditions:
-- ASM/DoS/FPS are provisioned.
-- BIG-IP device is experiencing a high CPU or I/O rate.
Impact:
High CPU Usage.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
Under certain conditions, on specific platforms, TMM does not follow best practices for TCP connections.
Conditions:
-- Platforms without HW based syn-cookie capability.
-- System DB variable connection.syncookies.algorithm value is set to "hardware."
Impact:
TMM does not follow best practices for TCP connections.
Workaround:
Set 'sys db connection.syncookies.algorithm value' to "software".
Component: Application Visibility and Reporting
Symptoms:
Under certain conditions, the AVRD application on BIG-IP may crash.
Conditions:
-- Bot Defense Profile is attached to the virtual server.
-- Mobile SDK is licensed and enabled.
Impact:
Instability in the avrd process. Statistics data can be lost.
Workaround:
None.
Component: TMOS
Symptoms:
In the Dashboard .csv file, all of the values in the first row are set to 'NaN'
Conditions:
This is encountered when loading the downloaded Dashboard .csv files with historical data from the GUI.
Impact:
The first row of the downloaded .csv from Dashboard shows all the values as 'NaN'.
Workaround:
None.
Component: Global Traffic Manager (DNS)
Symptoms:
GTM/DNS monitors based on the TCP protocol may fail to find the configured receive string in the server's response, causing the monitored service to be incorrectly marked down.
Conditions:
This issue occurs when the server's response spans multiple packets (for example, when the response is particularly large or includes dynamically generated content delivered in chunks).
Impact:
A service is incorrectly marked down. This can cause the BIG-IP GTM/DNS to return a suboptimal answer or no answer at all to DNS queries.
Workaround:
This issue can be worked around by ensuring your server returns a response to the BIG-IP GTM/DNS's monitor that fits in one packet.
Component: Application Security Manager
Symptoms:
When entering Live Update page, sometimes Update Available notification is shown twice.
Conditions:
You first enters Live Update page.
Impact:
Notification is shown twice.
Workaround:
None.
Component: Application Security Manager
Symptoms:
Some ASM attack signatures are not matching as expected on a cookie value.
Conditions:
Matching enabled for specific attack signatures.
Impact:
False negative. A signature does not match and an attack is getting through.
Workaround:
There is no valid workaround except creating custom signatures for cookies from all these signatures.
Component: TMOS
Symptoms:
Network Map color scheme is not compliant with Section 508 of the Rehabilitation Act (section 508). There is no clear difference between a green/active node and the blue/square items. With the new system colors and flat shapes, the icons are nearly identical. Other than shape (circle vs. square), the new colors appear identical; the blue and green are nearly appearing as one color.
Conditions:
Accessing Network Map from GUI via Local Traffic :: Network Map.
Impact:
There is no clear color difference between a green/active node icon and the blue/square icon.
Workaround:
None.
Component: Global Traffic Manager (DNS)
Symptoms:
The GTM/DNS HTTPS monitor may fail to find the configured receive string in a HTTP response, causing the monitored service to be incorrectly marked down.
Conditions:
This issue occurs when all of the following conditions are true:
-- The server being monitored performs an unclean shutdown of the SSL session (the underlying TCP connection is closed without first issuing a close notify alert at the SSL level).
-- The server's HTTP response does not terminate with a newline.
Impact:
A service is incorrectly marked down. This can cause the BIG-IP GTM/DNS to return a suboptimal answer or no answer at all to DNS queries.
Workaround:
This issue can be worked around by performing any one of the following actions:
-- Ensure the server issues a close notify alert before it closes the underlying TCP connection.
-- Ensure the server's HTTP response ends with a newline.
Component: Application Security Manager
Symptoms:
In the GUI, for the Application Security Administrator role, when you create a new ASM policy, the Policy Type is greyed out and the parent policy cannot be created
Conditions:
-- Create user account with the Application Security Administrator user role.
-- Use that account to logon to the GUI and try to create/edit the parent policy.
Impact:
The following actions are restricted to accounts with roles Application Security Administrator:
-- Create/Edit parent policy.
-- Edit Inheritance Settings for parent policy.
-- Clone Policy, selecting policy type is disabled.
Workaround:
There are two possible workarounds:
-- Have the Administrator or Resource Administrator create a parent policy instead of the Application Security Administrator.
-- Create parent policy using tmsh or REST call.
Component: Application Security Manager
Symptoms:
ASM processes some requests longer than usual and can cause some latency when passing requests to the backend server.
Conditions:
Content profile parsing is enabled in ASM
Impact:
Latency can be noted when passing requests to the backend server
Workaround:
ASM is processes requests as expected
Component: TMOS
Symptoms:
An authenticated iControl REST user may gain elevated privileges using unspecified means.
Conditions:
iControl REST users with non-admin privileges configured
Impact:
A lower privileged iControl REST user may be able to execute functions requiring a higher privilege level.
Component: Global Traffic Manager (DNS)
Symptoms:
-- RESOLV::lookup returns an empty string.
-- TMM might crash.
Conditions:
An iRule runs RESOLV::lookup targeting the query toward a local virtual server. For instance:
RESOLV::lookup @/Common/my_dns_virtual www.example.com
Impact:
RESOLV::lookup does not return the expected result;
tmm might crash. Traffic disrupted while tmm restarts.
Workaround:
In the RESOLV::lookup command, replace the name of the virtual server with its IP address, or the IP address of an external DNS server.
For instance, if /Common/my_dns_virtual has destination 192.0.2.53:53:
instead of this: RESOLV::lookup @/Common/my_dns_virtual
use this: RESOLV::lookup @192.0.2.53
Component: Local Traffic Manager
Symptoms:
QUIC sometimes rejects valid 0RTT packets.
Conditions:
-- QUIC enabled.
-- The Connection ID length assigned by the client for the server's CID does not match what the server assigned.
Impact:
QUIC drops 0RTT packets. Lost 0RTT packets increase latency.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
When AAM is provisioned the wocplugin resource allocation may fail, which could result in a restart loop of the plugin. This renders the AAM module nonfunctional.
Conditions:
Application Acceleration Manager (AAM) is provisioned
Impact:
AAM is not functional
Workaround:
The wocplugin is now correctly provisioned and runs without restarts.
Component: TMOS
Symptoms:
A configuration error occurs during upgrade due to custom partition-associated Draft folder, which exists in configuration file after deleting a custom partition.
Configuration error: Can't associate folder (/User/Drafts) folder does not exist.
Conditions:
This occurs in the following scenario:
1. Create Partition.
2. Create Policy under that partition.
3. Delete Policy.
4. Delete Partition.
5. Upgrade.
Impact:
Upgrade fails when a Drafts folder exists under the custom partition folder, if the custom partition is deleted.
Workaround:
Manually remove the stale folders in the configuration file, or use a script to remove them.
Component: Local Traffic Manager
Symptoms:
The BIG-IP system supports QUIC and HTTP/3 draft-27 and draft-28. IETF has released draft-29.
Conditions:
Browser requests draft-29.
Impact:
Connection downgrades to an older version, or fails if the browser cannot downgrade.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
Some HTTP/2 response headers may be added to the dynamic header table even if this header is already stored in the table. Instead of subsequent responses using the correct dynamic header table index, these headers may be continually seen as being incrementally indexed.
Conditions:
-- HTTP/2 clientside profile.
-- Concurrent HTTP/2 responses contain headers.
Impact:
Select HTTP/2 response headers may fail to use the dynamic header table index. These headers are incrementally indexed on subsequent responses instead of using the existing table index.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
If iRulesLX is provisioned the /usr mount points are mounted as read-write. This causes the installation of an image to fail.
tmsh show software status will report the status for the target volume:
Could not access configuration source.
Conditions:
-- iRulesLX is provisioned.
-- The /usr mount points are mounted as read-write.
-- Attempt an installation or upgrade.
Impact:
Unable to upgrade or more generally install an image on a new or existing volume.
Workaround:
Re-mount /usr as read-only:
mount -o remount,ro /usr
Component: TMOS
Symptoms:
When you navigate to the traffic class creation page by clicking Create button in the Traffic Class list page, Chinese characters are displayed with multiple question marks.
Conditions:
This is encountered when creating a new Traffic Class.
Impact:
Multi-byte characters are displayed incorrectly.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
Increased CPU usage due to zombie TCP flows rearming TCP Keep Alive timer continuously and unnecessarily.
Conditions:
-- A large number of zombie flows exists.
-- TCP Keep Alive timer is rearmed aggressively for zombie flows with very small idle_timeout (0) value.
-- TCP Keep alive timer keeps expiring and is rearmed continuously.
Impact:
Continuous rearming results in consuming CPU resources unnecessarily.
Workaround:
None.
Component: TMOS
Symptoms:
Using crontab to automatically backup UCS file by scheduling cronjobs fails due to SELinux permissions. The failure produces the following error:
Unexpected Error: UCS saving process failed.
Conditions:
This is encountered when 'tmsh save sys ucs' is executed through a cronjob.
Impact:
UCS file is not successfully saved and backup fails.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
The value of the QUIC logging level is erroneously shown as 'Error' in the GUI.
Conditions:
Set tmm.quic.log.level to 'Info' or 'Critical' in TMSH.
Impact:
Misleading log level displayed in the GUI.
Workaround:
Use TMSH to set and view values for tmm.quic.log.level.
Component: Access Policy Manager
Symptoms:
USB is unavailable after opening VMware View Desktop.
Conditions:
1. Secure Tunnel disabled on VCS
2. Launch view virtual desktop via native view client from an APM webtop or from the View client
Impact:
USB is unavailable after opening VMware View Desktop
Workaround:
None.
Component: Application Security Manager
Symptoms:
The Security - Options section in the left navigation menu is visible for only for user accounts configured with the Administrator role.
Conditions:
You logged in as a user configured with a role other than Administrator.
Impact:
No direct access to many settings that are available only for user account configured with the Administrator role.
Workaround:
Direct links to the pages work for those with the appropriate roles.
Component: Local Traffic Manager
Symptoms:
The following UDP statistics are missing:
-- bufdropdgram
-- maxrate_conns
-- maxrate_cur_conns
-- sendbuf_cur_bytes
-- queue_dropped_bytes
Conditions:
Viewing UDP statistics.
Impact:
Unable to view these UDP statistics.
Workaround:
None.
Component: Application Security Manager
Symptoms:
ASM policy has the option 'Learn only from non-bot traffic' enabled, but the Policy Builder detects that the client is a bot, and therefore does not issue learning suggestions for the traffic.
Conditions:
-- ASM policy is enabled via LTM policy.
-- ASM policy configured to learn only from non-bot traffic.
This applies to complex policies, and in some configurations may happen also when a simple policy is enabled via LTM policy.
Impact:
No learning suggestions.
Workaround:
Disable the option 'Learn only from non-bot traffic' on the ASM policy.
Component: iApp Technology
Symptoms:
Under certain conditions, iAppsLX does not follow current best security practices.
Conditions:
iAppsLX REST access by an authenticated administrative user
Impact:
iAppsLX does not follow current best security practices.
Workaround:
iAppsLX now follows current best security practices.
Component: Local Traffic Manager
Symptoms:
On certain platforms, after upgrade to BIG-IP v16.0.0, you will not be able to boot back into an earlier software version. Contact F5 Support for the reversion process if this is required.
Conditions:
-- Using the following platforms:
+ i5820-DF / i7820-DF
+ 5250v-F / 7200v-F
+ 10200v-F
+ 10350v-F
-- Upgrade the software to v16.0.0.
-- Attempt to roll back to a previous version.
Impact:
Cannot boot into a previous version. Contact F5 Support for the reversion process if this is required.
Workaround:
None.
Behavior Change:
On certain platforms, after upgrade to BIG-IP v16.0.0, you will not be able to boot back into an earlier software version. Contact F5 Support for the reversion process if this is required.
The particular platforms are:
+ i5820-DF / i7820-DF
+ 5250v-F / 7200v-F
+ 10200v-F
+ 10350v-F
Component: TMOS
Symptoms:
TMM crashes with an assert; Drop assertion similar to the following:
notice panic: ../dev/ndal/ndal.c:758: Assertion "pkt length cannot be greater than MAX_PKT_LEN" failed.
Conditions:
-- Xnet driver is used in BIG-IP Virtual Edition (VE).
-- TMM tries to send oversize packets.
Impact:
TMM crashes. Traffic disrupted while tmm restarts.
Workaround:
None.
Component: Device Management
Symptoms:
Under certain conditions, iControl REST endpoint return the request contents in the response, potentially disclosing sensitive information.
Conditions:
-iControl REST request processed by an authenticated administrative user.
-Undisclosed environmental conditions
Impact:
Potential disclosure of information from REST requests.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
The BIG-IP system supports QUIC and HTTP/3 draft-25 and draft-27. IETF has released draft-28.
Conditions:
Browser requests draft-28.
Impact:
Connection downgrades to an older version, or fails if the browser cannot downgrade.
Workaround:
None.
Component: Advanced Firewall Manager
Symptoms:
TMM core resulting in potential loss of service.
Conditions:
Attaching and deleting the vector to a DoS profile multiple times while the traffic is ongoing.
Impact:
Traffic disrupted while tmm restarts.
Workaround:
None.
Component: TMOS
Symptoms:
After SPF/IA calculation gets suspended, it might enter a state where it never fires again.
Conditions:
SPF/IA calculation gets suspended;
This occurs for various reasons; BIG-IP end users have no influence on it occurring.
Impact:
OSPF routes are visible in the OSPF database, but not installed in the routing table.
Workaround:
Restart the routing daemons:
# bigstart restart tmrouted
Running this command allows you to recover from this situation, but does not prevent the event from reoccurring.
If due to a topology, SPF/IA calculation suspension occurs again after a restart, this workaround essentially has no effect.
Component: Local Traffic Manager
Symptoms:
HTTP/3 throughput is poor.
Conditions:
Virtual Server configured with an HTTP/3 profile.
Impact:
Performance might be severely degraded.
Workaround:
There is no alternative other than not using the HTTP/3 profile.
Component: Access Policy Manager
Symptoms:
Cluster failover, tmm restart, and tmm killed due to missed heartbeats. tmm crash
Conditions:
TMM is under load due to heavy traffic while MCP attempts to configure per-request policy. This can be caused by a modification to the policy or one of its agents, or by a restart of the TMM.
Impact:
Traffic disrupted while tmm restarts.
Workaround:
When making changes to per-request policies, use a scheduled maintenance window so that impact to traffic is minimized.
Component: Local Traffic Manager
Symptoms:
Under certain conditions, TMM may consume excessive resources when processing forwarding flows while AFM is provisioned.
Conditions:
- AFM provisioned
- Virtual is CMP-disabled
Impact:
Excessive resource consumption, potentially leading to a failover event.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
TMM crashes.
Conditions:
-- Virtual server has HTTP and HTTP Router profiles attached.
-- Certain scenarios where the proxy goes into passthrough mode.
This was encountered during internal testing of a certain iRule configurations.
Impact:
Traffic disrupted while tmm restarts.
Workaround:
None.
Component: Global Traffic Manager (DNS)
Symptoms:
Under certain conditions, TMM may crash with a SIGABRT while processing DNS traffic.
Conditions:
Undisclosed series of DNS requests.
Impact:
TMM crash leading to a failover event.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
TMM crashes.
Conditions:
-- Virtual server has HTTP and HTTP Router profiles attached to it.
-- Certain scenarios where the proxy goes into passthrough mode.
Impact:
Traffic disrupted while tmm restarts.
Workaround:
None.
Component: Application Visibility and Reporting
Symptoms:
AVR logrotate reports errors in /var/log/avr:
error: error creating output file /var/log/avr/avrd.log.1: File exists
/var/log/avr/avrd.log will remain unchanged
/var/log/avr/avrd.log.1 will remain unchanged
Conditions:
Files ending with .1 exist in the log directory.
Impact:
Logrotate does not work. This might fill the disk with logs over time.
Workaround:
Remove or rename all of the .1 log files.
Component: TMOS
Symptoms:
Memory leak in BWC calculator.
Conditions:
When the HSL log publisher is attached to the BWC::Measure instance in the Bandwidth policy.
Impact:
A memory leak occurs.
Workaround:
None.
Component: TMOS
Symptoms:
Incorrect totals for New Flows under Security :: Debug :: Flow Inspector :: Get Flows.
Conditions:
Viewing New Flows under Security :: Debug :: Flow Inspector :: Get Flows.
Impact:
Calculation mistake in the GUI: shows 8 times the actual values, for example:
Packets In 2 shows as 016 in the GUI
Packets Out 0 shows as 8 in the GUI
Workaround:
View statistics in tmsh.
Component: Global Traffic Manager (DNS)
Symptoms:
A TMM daemon logs a SIGSEGV error, halts, and then be restarted.
Conditions:
-- A BIG-IP system configured to perform DNS/GTM Global Server Load Balancing.
-- High Speed Logging (HSL) is configured.
-- Multiple HSL destinations are configured.
-- The enabled HSL settings include 'replication'.
-- At least one HSL destination is up.
-- At least one HSL destination is down.
-- A pool resource changes state from up to down.
Impact:
Traffic disrupted while tmm restarts.
Workaround:
Configure HSL with only a single log destination.
Component: Access Policy Manager
Symptoms:
Under certain conditions, the APM Webtop does not follow current best practices.
Conditions:
-APM access policy is configured with webtop enabled.
Impact:
APM Webtop does not follow current best practices.
Workaround:
None.
Component: Global Traffic Manager (DNS)
Symptoms:
Under certain conditions, zxfrd may consume excessive resources.
Conditions:
-- Provision GTM (Global Traffic DNS).
-- Create one or more zones in the GUI via DNS::Zones::Zones::Zone List.
Impact:
Excessive resources consumption, potentially leading to failover event.
Workaround:
None.
Component: TMOS
Symptoms:
The result of guest OS customization depends on the DHCP state on the management (mgmt) interface and the applied customization profile (i.e., IPv4 only, IPv4 and IPv6, or IPv6 with IPv4 prompt).
By default, DHCP is enabled on the management interface.
During configuration, you can customize only one IPv4 or one IPv6 address in a dual stack environment.
Conditions:
Applying a customization profile to VMware VM in a dual stack environment.
Impact:
You can only partially customize the mgmt interface IP profiles for VMware VMs in a dual stack environment.
Workaround:
Configure the mgmt interface addresses using the config script.
Component: TMOS
Symptoms:
Cannot clone Azure marketplace instances because cloned instances do not properly retrieve publisher and product code from the metadata service.
Conditions:
Applies to any Azure marketplace instance.
Impact:
Cannot clone Azure marketplace instances.
Workaround:
None.
Component: Application Security Manager
Symptoms:
When a Cloud Auto Scaling deployment is set up using F5's Auto Scale Template, and ASM Live Update is configured with Automatic Download enabled, the configuration may be overwritten during a scale out event when a new host joins the sync cluster. This is caused by a config sync from the new device to the master device, before the master has a chance to sync the configuration to the new device, causing the configuration in the master device to be overwritten.
Conditions:
-- Using F5's Auto Scaling template.
-- Auto Scale script is configured with --block-sync (which is the default).
-- ASM Live Update is configured with Automatic Download enabled.
-- A scale out event occurs.
Impact:
Configuration of all devices in the Auto Scale group is overwritten.
Workaround:
Disable ASM Live Update Automatic Download.
This can be done by disabling the liveupdate.autodownload DB variable using the onboard.js script, and adding '-d liveupdate.autodownload:disable'.
For example:
/usr/bin/f5-rest-node /config/cloud/aws/node_modules/@f5devcentral/f5-cloud-libs/scripts/onboard.js --log-level silly --signal ONBOARD_DONE -o /var/log/cloud/aws/onboard.log --host localhost --port 8443 -d tm.tcpudptxchecksum:software-only -d liveupdate.autodownload:disable --ping
-d tm.tcpudptxchecksum:software-only -d liveupdate.autodownload:disable
Component: Service Provider
Symptoms:
The GenericMessage filter does not have a configurable limit to the number of messages that can be received.
Conditions:
If a message is waiting for an asynchronous iRule operation during a GENERICMESSAGE_INGRESS or GENERICMESSAGE_EGRESS iRule event, new messages are placed in either the ingress or egress queue. As the number of messages increase, more memory is required.
Impact:
If too many messages are queued, the system may exceed an internal count which could lead to a core.
Workaround:
None.
Component: Access Policy Manager
Symptoms:
Under certain conditions APMD may crash while processing certain parameters.
Conditions:
-APM active.
-Pre-inspection checks active.
Impact:
APMD crash, leading to a failover event
Workaround:
None.
Component: Application Security Manager
Symptoms:
Disabling ASM Main Cookie/Domain Cookie hashing in a Policy's Learning and Blocking Setting with 'Never (wildcard only)' does not stop the ASM Main Cookie from continuing to hash server-provided cookies.
Conditions:
-- ASM enabled.
-- Learning mode enabled for Policy.
-- Learn New Cookies set to 'Never (wildcard only)' instead of default 'Selective'.
Impact:
A sufficient number of ASM Main Cookies and/or a sufficiently large number of cookies for each ASM Main cookie to hash can result in the HTTP header becoming prohibitively large, causing traffic to be refused by the server.
Workaround:
Disable Learning mode for the Policy disables Cookie hashing.
Note: This affects all learning, not just Cookie hashing.
Component: Access Policy Manager
Symptoms:
Over time, memory is allocated by the TMM processes for use as 'xdata' buffers, yet this memory is never de-allocated; it is leaked and becomes unusable. Eventually a disruption of service occurs.
Conditions:
-- The BIG-IP system has been running for 8 weeks or longer without a system restart.
-- The BIG-IP system's internal risk-policy subsystem (used by the security feature modules) has not been configured to communicate with an external risk-policy server.
-- In a vCMP configuration, the BIG-IP 'host' instance is always susceptible, since no security features can be configured in its context.
Impact:
Traffic disrupted while tmm restarts.
Workaround:
None.
Component: Application Security Manager
Symptoms:
Security :: Bot Defense : Bot Defense Profiles page loading takes a long time when there are profiles configured with hundreds of virtual servers. For example: a configuration with 750 virtual servers takes about 40 seconds to load the page. Configuration with 1300 virtual servers takes more than 70 seconds.
Conditions:
At least one Bot profile attached to hundreds of virtual servers. For 750 and more virtual servers attached the slow loading is significant.
Impact:
Bot Defense list page loading time can take more than 30 seconds.
Workaround:
None.
Component: Application Visibility and Reporting
Symptoms:
In AVR pool-traffic report, 'server-concurrent-conns' reports a larger value than 'server-max-concurrent-conns'.
Conditions:
This is encountered when viewing the pool-traffic report.
Impact:
Incorrect stats reported in the pool-traffic report table
Workaround:
In /etc/avr/monpd/monp_tmstat_pool_traffic_measures.cfg, change the formula of server_concurrent_connections:
From this:
formula=round(sum(server_concurrent_conns),2)
Change it to this:
formula=round(sum(server_concurrent_conns)/count(distinct time_stamp),2)
Component: TMOS
Symptoms:
Error during config load due to custom partition associated Draft folder exists after deleting partition.
01070734:3: Configuration error: Can't associate folder (/User/Drafts) folder does not exist
Unexpected Error: Loading configuration process failed.
Conditions:
Create draft policy under custom partition
Impact:
Impacts the software upgrade.
Workaround:
Remove the Draft folder config from bigip_base.conf or use command "tmsh delete sys folder /User/Drafts" followed by "tmsh save sys config" after removing partition.
Component: TMOS
Symptoms:
The ospfd process generates a core.
Conditions:
-- IA routes.
-- OSPF is in FULL/DR state.
Impact:
An OSPF daemon generates a core, potentially losing routing information and OSPF dynamic routes for a moment while the ospfd daemon restarts.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
When a virtual server is created, GARPs are not sent out.
Conditions:
-- Creating a new virtual server.
Impact:
Traffic could be impacted if other systems have the virtual server address already in their ARP caches.
Workaround:
After creating the virtual server, disable and re-enable the ARP setting on the corresponding virtual address.
Component: Application Security Manager
Symptoms:
As a fix to ID879777 (https://cdn.f5.com/product/bugtracker/ID879777.html), when navigating to a related domain using Safari, requests might be blocked.
Conditions:
- Using Bot Defense profile, with "Cross Domain Requests" mode "Validate Upon Request"
- BIG-IP version containing fix of ID879777 (16.0 and higher or EngHF)
- Surfing the site using Safari browser.
Impact:
Some requests might be blocked.
Workaround:
None.
Component: Application Security Manager
Symptoms:
Brute Force Protection (BFP) uses a hash table to store counters of failed logins per IPs and usernames.
There is a separate hash table for each virtual server.
When the hash table is fully utilized and new entries need to be added, the LRU entry is being removed.
This scenario may cause mitigated entries to keep getting removed from the hash table by new entries.
Conditions:
There is a separate hash table for each virtual server, and its size is controlled by the external_entity_hash_size internal parameter.
When it is set to 0, the size is determined automatically based on system memory.
Otherwise, it is the maximum size of the hash tables together, then divided into the number of virtual servers which have traffic and BFP enabled.
In case of the latter, there might be a chance that with too many virual servers the hash table may reach it's maximum capacity.
Impact:
Mitigated entries that keep getting removed from the hash table by new entries, may result in attacks not getting mitigated.
Workaround:
Mitigated entries are kept in the hash table.
Component: Application Security Manager
Symptoms:
Under certain conditions, resources for Brute Force Protection must be manually scaled by administrators to provide full protection.
Conditions:
-- Many virtual server (hundreds) that have web application protection with brute force protection enabled.
-- Numerous failed login requests coming to all virtual servers all the time.
Impact:
Administrators must manually change the hash size upon need instead of relying on the automatic configuration.
Workaround:
Set the internal parameter external_entity_hash_size to 0 to allow automatic recalculation of the correct value.
Component: Application Security Manager
Symptoms:
Brute Force Protection (BFP) uses a hash table to store counters of failed logins per IP addresses and usernames. There is a separate hash table for each virtual server. When the hash table is fully utilized and new entries need to be added, the LRU entry is being removed without logging a warning.
Conditions:
This can be encountered when Brute Force Protection is enabled and the hash table reaches its maximum capacity.
Impact:
No alert is sent when entries are evicted.
Workaround:
None.
Component: Service Provider
Symptoms:
GTP message parsing fails and log maybe observed as below:
GTP:../modules/hudfilter/gtp/gtp_parser.c::242 (Failing here. ).
GTP:../modules/hudfilter/gtp/gtp_parser_ver_2.c::153 (Failing here. ).
GTP:../modules/hudfilter/gtp/gtp_parser.c::103 (Failing here).
Conditions:
- GTP profile is applied to virtual or GTP::parse command is used
- GTP message contains IE (information element) which is larger than 2048 bytes
Impact:
- message parsing fails, traffic maybe interupted
Component: Local Traffic Manager
Symptoms:
Under certain conditions, APM may consume excessive resources while processing VPN traffic
Conditions:
-APM provisioned
-VPN clients connected
Impact:
Excessive resources consumption, potentially leading to a TMM crash and failover event.
Workaround:
None.
Component: Application Security Manager
Symptoms:
After activating a FIPS 140-2 license on a device and rebooting, the device fails to boot.
Conditions:
-- Device is licensed for FIPS 140 mode
-- A critical system file is missing
Impact:
System halts during boot because of sys-eicheck.py failure.
Workaround:
Prior to rebooting into FIPS 140-2 mode, ensure that there are no missing critical files by running the sys-eicheck command.
If the missing files are due to missing signature update files:
- Manually upload the missing images in System ›› Software Management : Live Update - this will ensure that the image is associated with an installation record.
Component: TMOS
Symptoms:
-- When an IPv6 address used in the command 'ipv6 static <addr> <gateway> fall-over bfd' exceeds 19 characters, it gets truncated.
-- IPv6 static BFD configuration entries go missing during a daemon restart.
Conditions:
IPv6 static BFD configuration.
Impact:
The IPv6 static BFD configuration does not persist during reloads.
-- The long IPv6 addresses get truncated.
-- The configuration is removed upon daemon restart.
Workaround:
None.
Component: TMOS
Symptoms:
The following SCTP iRule commands:
-- SCTP::mss
-- SCTP::ppi
-- SCTP::collect
-- SCTP::respond
-- SCTP::client_port
-- SCTP::server_port
Are unavailable in the following MRF iRule events:
-- GENERICMESSAGE_EGRESS
-- GENERICMESSAGE_INGRESS
-- MR_EGRESS
-- MR_INGRESS
Attempts to use these commands in these events result in errors similar to:
01070151:3: Rule [/Common/sctp_TC] error: /Common/sctp1: error: [command is not valid in current event context (GENERICMESSAGE_EGRESS)][SCTP::ppi 46].
Conditions:
-- Using MRF and SCTP.
-- Using the specified set of iRule commands within the listed iRule events.
Impact:
Unable to use these iRule commands within these iRule events.
Workaround:
None.
Component: Global Traffic Manager (DNS)
Symptoms:
When you use the 'Remove' button to remove a member from a WideIP, the member is removed from all WideIPs.
Conditions:
Use the 'Remove' button.
Impact:
Unintended configuration changes via GUI.
Workaround:
Use the 'Manage' button, rather than the 'Remove' button.
Component: Advanced Firewall Manager
Symptoms:
The fw_zone_stat 'Hits' field does not reflect the current stats.
Conditions:
When the firewall rule has multiple VLANs defined as destinations (in a zone).
Impact:
The counter for all VLANs does not hit : fw_zone_stat. The corresponding stat value does not increment.
Workaround:
None.
Component: Application Security Manager
Symptoms:
No parent entity appears in an ASM Policy Builder suggestion to add to the policy a predefined-filetype to the allowed filetypes list.
Conditions:
The issue is encountered when filetypes are configured with learning mode which allows new filetypes to be added to the policy. Relevant learning modes to this issue are: Always, Selective and Compact.
Impact:
No parent entity appears in the sugestion.
Workaround:
None.
Component: TMOS
Symptoms:
When you try to log into the GUI via the management IP, you see only a single gray bar displayed in the middle of the window.
Conditions:
-- A GUI session expired while you were logged on.
-- The partition on which the GUI session expires is deleted.
-- You log on again.
Impact:
GUI becomes unresponsive.
Workaround:
Restart tomcat via SSH:
# bigstart restart tomcat
Component: Local Traffic Manager
Symptoms:
HTTP::has_responded is not detected in an iRule when the request comes across via HTTP/2. Instead, HTTP::has_responded always return the value 'false'.
Conditions:
-- HTTP/2 profile.
-- iRule containing the command HTTP::has_responded.
Impact:
Calls to HTTP::respond or HTTP::redirect are not correctly identified by HTTP::has_responded when using HTTP/2.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
If a client connection closes before finishing client ssl handshake, in some cases BIG-IP ssl does not close and connection remains until idle timeout.
Conditions:
Client ssl handshake and client FIN must arrive while BIG-IP server ssl finished is in crypto.
Impact:
Some ssl client connection remain until idle timeout.
Component: Application Security Manager
Symptoms:
REST framework not available due to Out of memory error
Conditions:
Long list of Live Update installations
Impact:
Live Update GUI is not responding.
Workaround:
1) Increase memory assigned to the Linux host: (value dependant on platform)
# tmsh modify sys db provision.extramb value 1000
2) Allow restjavad to access the extra memory:
# tmsh modify sys db restjavad.useextramb value true
3) Save the config:
# tmsh save sys config
4) The re-provisioning will trigger a restart of the services. Wait until the unit is online again.
5) Increase the restjavad maxMessageBodySize property:
# curl -s -f -u admin: -H "Content-Type: application/json" -d '{"maxMessageBodySize":134217728}' -X POST http://localhost:8100/mgmt/shared/server/messaging/settings/8100 | jq .
"maxMessageBodySize": 134217728,
"localhostRestnodedConnectionLimit": 8,
"defaultEventHandlerTimeoutInSeconds": 60,
"minEventHandlerTimeoutInSeconds": 15,
"maxEventHandlerTimeoutInSeconds": 60,
"maxActiveLoginTokensPerUser": 100,
"generation": 6,
"lastUpdateMicros": 1558012004824502,
"kind": "shared:server:messaging:settings:8100:restservermessagingpoststate",
"selfLink": "https://localhost/mgmt/shared/server/messaging/settings/8100"
Ensure the command returns output showing the limit has been increased (as shown above).
6) Reboot the unit.
Component: Local Traffic Manager
Symptoms:
https://support.f5.com/csp/article/K20105555
Conditions:
https://support.f5.com/csp/article/K20105555
Impact:
https://support.f5.com/csp/article/K20105555
Workaround:
https://support.f5.com/csp/article/K20105555
Component: Access Policy Manager
Symptoms:
https://support.f5.com/csp/article/K20105555
Conditions:
https://support.f5.com/csp/article/K20105555
Impact:
https://support.f5.com/csp/article/K20105555
Workaround:
https://support.f5.com/csp/article/K20105555
Component: TMOS
Symptoms:
While loading configuration from the file with IMISH ('imish -f <f_name>'), the newline character gets copied at the end of each line which causes problems with commands containing regex expressions.
In particular, this affects the bigip_imish_config Ansible module.
Conditions:
Loading a config with 'imish -f <f_name>' commands.
Note: This command is used with the bigip_imish_config Ansible module.
Impact:
Regex expressions are not created properly.
Workaround:
You can use either of the following workarounds:
-- Delete and re-add the offending commands using the imish interactive shell.
-- Restart tmrouted:
bigstart restart tmrouted
Component: Application Security Manager
Symptoms:
You are unable to define Maximum Query String Size and Maximum Request Size fields for Splunk Logging Format in the GUI.
Conditions:
This is encountered when configuring the Splunk Logging Format in the GUI
Impact:
You are unable to define Maximum Query String Size and Maximum Request Size fields for Splunk Logging Format in the GUI.
Workaround:
Use tmsh to define the maximum query string size and maximum request size. For more information, see the tmsh command reference for the security log profile at https://clouddocs.f5.com/cli/tmsh-reference/v14/modules/security/security-log-profile.html
Component: Application Security Manager
Symptoms:
The 'Add' button is grayed out on Security :: Application Security :: Microservices, where in previous version, minimal microservice creation/modification was available.
Conditions:
-- Navigate to Security :: Application Security :: Microservices.
-- Attempt to add a microservice.
Impact:
Cannot add or modify a microservice in this version, where it was available in previous versions.
Workaround:
None.
Component: Advanced Firewall Manager
Symptoms:
BDoS Signature mitigated in software.
Conditions:
IP packets size metric in BDoS signature.
Impact:
BDoS Signature with IP packet size metric mitigated only in software for IPv6 packets.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
WebSocket connection hangs on the clientside if the serverside WebSocket payload is small and received in the same TCP packet with server HTTP response.
Conditions:
-- Virtual contains HTTP and WebSocket filters.
-- HTTP response and a small WebSocket payload is received in the same TCP packet from the server.
-- Small WebSocket payload is not delivered on the clientside.
Impact:
-- WebSocket connection hangs.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
Running iRule commands while suspending iRule commands that are running can lead to a crash.
Conditions:
-- Running iRule commands.
-- iRule commands that suspend iRules are running.
For more information on the conditions that trigger iRule suspend, see K12962: Some iRules commands temporarily suspend iRules processing, available at https://support.f5.com/csp/article/K12962.
Impact:
Tmm crashes. Traffic disrupted while tmm restarts.
Workaround:
None.
Component: Fraud Protection Services
Symptoms:
An error exists in datasyncd.log:
DATASYNCD|ERR |Mar 13 12:47:54.079|16301| datasyncd_main.c:1955|tbl_gen_state_machine: cannot start the generator for table CS_FPM
Conditions:
Upgrades from version 13.x to 14.0.0 or higher.
Impact:
FPS has a maximum of ~990 rows instead of 1001, and there are errors in datasyncd.log. However, the upgrade completes normally, and the system operates as expected.
Workaround:
These are benign error messages that you can safely ignore. Upgrade completes successfully, and the system operates as expected.
If you prefer, however, you can perform a clean install instead instead of upgrading. This has an impact on your configuration, as that information will be lost when you do a clean install.
Component: TMOS
Symptoms:
When a URL containing a query string is added to an anti-fraud profile, the BIG-IP config load fails:
010719d8:3: Anti-Fraud URL '/url\?query=string' is invalid. Every protected URL should be a valid non-empty relative path specified in lower case in the case insensitive Anti-Fraud profile '/Common/antifraud'.
Unexpected Error: Loading configuration process failed.
Conditions:
Adding a query string to a URL for an anti-fraud profile.
Impact:
After a BIG-IP config save, loading of new bigip.conf fails.
Workaround:
Follow this procedure:
1. Remove the escaping characters \ (backslash) for ? (question mark) in the bigip.conf file.
2. Load the configuration.
Component: TMOS
Symptoms:
When a bandwidth control policy is applied on a virtual server, the BIG-IP system does not retransmit unacknowledged data segments, even when the BIG-IP system receives a duplicate ACK.
Both static bandwidth control policies and dynamic bandwidth control policies can cause the problem.
Conditions:
This issue occurs when both of the following conditions are met:
-- Virtual server configured with a bandwidth control policy.
-- Standard type of virtual server.
Impact:
The BIG-IP system does not retransmit unacknowledged data segments.
Workaround:
None.
Component: Service Provider
Symptoms:
MRF SIP load balancing does not support the urn URI protocol type.
Conditions:
-- Using MRF SIP in LB mode.
-- Clients are using the urn protocol type in their URIs.
Impact:
SIP messages with urn URIs are rejected.
Component: TMOS
Symptoms:
Mcpd errors are found in /var/log/ltm:
-- err mcpd[16068]: 01070734:3: Configuration error: MCPProcessor::initializeDB: basic_string::_S_create
-- err mcpd[16068]: 01070596:3: An unexpected failure has occurred, basic_string::_S_create, exiting...
-- err mcpd[16978]: 01b50049:3: FipsUserMgr Error: Master key load failure.
Conditions:
-- Restart MCPD (e.g. by rebooting the BIG-IP)
-- This affects BIG-IP v15.1.0.4 running on all platforms with the exception of the following:
+ i5820-DF / i7820-DF
+ 5250v-F / 7200v-F
+ 10200v-F
+ 10350v-F
Impact:
MCPD does not restore the configuration from its binary database, but instead re-reads the text config files (bigip.conf, et al.).
Workaround:
None.
Component: TMOS
Symptoms:
The Georedundancy traps introduced in 15.0.1.2 with trap IDs in the F5 enterprise MIB of .1.3.6.1.4.1.3375.2.4.0.206 to .1.3.6.1.4.1.3375.2.4.0.211 should have been numbered from .1.3.6.1.4.1.3375.2.4.0.212 to .1.3.6.1.4.1.3375.2.4.0.217
Conditions:
When 15.0.1.2 is upgraded to 15.1.0 then the traps would be renumbered.
Impact:
This may be confusing for SNMP clients expecting specific trap IDs.
Workaround:
None.
Behavior Change:
New traps were introduced in 15.0.1.2 for Georedundancy with previously assigned trap numbers. These traps will be renumbered when you upgrade 15.0.1.2 to 15.1.0.
The Georedundancy traps introduced in 15.0.1.2 with trap IDs in the F5 enterprise MIB of .1.3.6.1.4.1.3375.2.4.0.206 to .1.3.6.1.4.1.3375.2.4.0.211 should have been numbered from .1.3.6.1.4.1.3375.2.4.0.212 to .1.3.6.1.4.1.3375.2.4.0.217
Component: TMOS
Symptoms:
When a full config sync is done to a device group with large number of partitions:
-- The config sync operation takes a long time to complete.
-- There is a spike in CPU usage on the device where config push is initiated.
-- The mcpd daemon is unresponsive to other daemons, such tmsh, GUI etc., as it is busy pushing the config sync.
-- iQuery connections are terminated due to high CPU utilization.
Conditions:
Full config sync on device with large number of partitions.
Impact:
The operation takes a long time to complete, minutes on a BIG-IP Virtual Edition (VE) configurations, and varies by platform and the size of the configuration. For example, config sync on a medium BIG-IP VE setup running v15.1.0.1 with 512 partitions takes ~3 minutes.
Impedes management of device as well as terminates iQuery connections to GTM/DNS devices.
Workaround:
Enable Manual Incremental Sync.
Component: Local Traffic Manager
Symptoms:
The source port is always changed in source-port preserve mode even if the original source port with the other parameters would hash to the same TMM.
Conditions:
This issue occurs when both of the following conditions are met:
-- The virtual server is configured with source-port preserve (the default).
-- The system uses one of the following hash configurations including IP addresses.
- Using RSS DAG as a default hash on BIG-IP Virtual Edition (VE) (Z100) or on 2000- and 4000-series devices.
- Configuring a VLAN's 'CMP Hash' setting to a non-default value.
- Using a special variable such as non-default udp.hash or tcp.hash.
Impact:
Applications relying on a specific, fixed source port might not work as expected.
Workaround:
Set source-port to preserve-strict.
Behavior Change:
Beginning with v16.0.0, the TM.PortFind.Src_Preserve BigDB variable introduced in v15.1.0 is no longer supported.
The source-port preserve setting now does best effort to preserve the source port.
Component: Advanced Firewall Manager
Symptoms:
Several SNMP OIDs need to be added to provide the total number of port block allocations (PBAs) and the percentage of PBAs that are available.
Conditions:
Attempting to retrieve total number of PBAs and percentage of PBAs that are available.
Impact:
Need to manually calculate the values.
Workaround:
Make manual calculations from the current stats or configuration.
Behavior Change:
The following new MIBs are now available:
F5-BIGIP-LOCAL-MIB::ltmLsnPoolStatTotalPortBlocks
F5-BIGIP-LOCAL-MIB::ltmLsnPoolStatPercentFreePortBlocksSnmp
F5-BIGIP-LOCAL-MIB::ltmFwNatDynamicPoolStatPbaTotalPortBlocks
F5-BIGIP-LOCAL-MIB::ltmFwNatDynamicPoolStatPbaPercentFreePortBlocksSnmp
Component: Access Policy Manager
Symptoms:
You can change the password even if there are different values in the fields 'New Password' and 'Confirm Password' or if 'Confirm Password' is empty.
Conditions:
-- Access Policy with 'Modern' customization.
-- Configure an access policy with 'Logon Page' and 'AD Auth' agents.
-- When forced to change passwords, type different values in 'New Password' and 'Confirm Password', or leave 'Confirm Password' empty.
Impact:
The system allows the password change, even though the 'New Password' and 'Confirm Password' do not match.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
Active Sflow receiver configuration may lead to all egress traffic getting dropped after TMM starts.
Conditions:
Enabled sflow receiver is configured.
Impact:
Egress traffic is dropped.
Workaround:
Disable Sflow receiver, save configuration, reboot.
Component: TMOS
Symptoms:
When a failover is triggered, the floating IP addresses do not migrate to the newly active device. In /var/log/auditd/audit.log, you see the following errors:
/var/log/auditd/audit.log:type=AVC msg=audit(1583426470.463:27492): avc: denied { read } for pid=26865 comm="curl" name="resolv.conf" dev="dm-5" ino=32804 scontext=system_u:system_r:f5config_failover_t:s0 tcontext=system_u:object_r:net_conf_t:s0 tclass=lnk_file
Conditions:
-- A failover event occurs.
-- oci-curl will be called when failover happens, which may be unable to read /etc/resolv.conf.
Impact:
Failover does not complete. Floating IP addresses do not move to the active device.
Workaround:
Run two commands:
tmsh modify sys db failover.selinuxallowscripts enable
setenforce 0
Impact of workaround: these commands disable SELinux policy enforcement.
Component: Advanced Firewall Manager
Symptoms:
There are several port block allocation (PBA) statistics that need to be added.
Conditions:
Attempting to retrieve total number of PBAs and percentage of PBAs that are available.
Impact:
Need to manually calculate the values.
Workaround:
Make manual calculations from the current stats or configuration.
Behavior Change:
The following new tmstat value is now available, in both 'tmctl fw_lsn_pool_pba_stat' and 'tmctl lsn_pool_pba_stat:
total_port_blocks
The relevant TMSH show commands have been updated to include these new values:
-- Total Port Blocks
-- Percent Free Port Blocks
Component: TMOS
Symptoms:
JSESSIONID, BIGIPAUTHCOOKIE, BIGIPAUTH can be seen in the browser's debugging page.
Conditions:
-- Accessing the BIG-IP system using the GUI.
-- Viewing the browser's stored cache information.
Impact:
HTTPS session information is captured/seen in the browser's local cache, cookie.
Note: The BIG-IP system does not display and/or return sensitive data in the TMUI. Content that is marked appropriately as sensitive is never returned, so it is never cached. Data that is cached for TMUI in the client browser session is not considered secret.
Workaround:
Disable caching in browsers.
Component: Application Security Manager
Symptoms:
When the URI and 'Referer' header share the same positional parameter, the 'Referer' positional parameter is not masked in logs.
Conditions:
Sending a request with positional parameter in URI and 'Referer' header.
Impact:
'Referer' header positional parameter value is not masked in logs.
Workaround:
None.
Component: Application Security Manager
Symptoms:
When importing a declarative policy with an updated template, the operation uses the old version of the template, which is saved on the machine.
Conditions:
Declarative policy is created from a user-defined template and then re-created after updating the template.
Impact:
The re-created declarative policy is based on the old template, which is saved without the changes that were made in the template.
Workaround:
Create new template and use it when recreating the declarative policy.
Component: Local Traffic Manager
Symptoms:
TMM cores in the HTTP proxy.
Conditions:
-- HTTP and HTTP Router profiles are configured on the virtual server.
-- The HTTP peer aborts the connection unexpectedly.
Impact:
Failover (in a DSC), or outage (standalone) as traffic is disrupted while tmm restarts.
Workaround:
None.
Component: TMOS
Symptoms:
Filenames with spaces in /config directory can cause upgrade/UCS load to fail because the im upgrade script that backs up the config, processes the lines in a file spec using white space characters. The number of spaces in the filename is significant because it determines how the process separates the name into various fields, including a path to the file, an md5sum, and some file properties (notably size). If the path contains white space, when the upgrade/UCS load process attempts to use a field, the operation encounters a value other than what it expects, so the upgrade/UCS load fails.
The file's content is also significant because that determines the md5sum value.
Although rarely occurring, a tangential issue exists when the sixth word is a large number. The sixth field is used to determine the amount of space needed for the installation. When the value is a very large number, you might see an error message at the end of the upgrade or installation process:
Not enough free disk space to install!
Conditions:
Filenames with spaces in /config directory.
Impact:
Upgrade or loading of UCS fails.
Workaround:
Remove the spaces in filenames and try the upgrade/UCS load again.
Component: Access Policy Manager
Symptoms:
TMM crash.
Conditions:
When user trying to access a URL with unique hostname in the current session.
Impact:
TMM crash. No access to the URL. Traffic disrupted while tmm restarts.
Workaround:
None.
Component: TMOS
Symptoms:
When creating virtual server or transport config containing both SCTP and Generic Message profile, it will fail with an error:
01070734:3: Configuration error: Profile(s) found on /Common/example_virtual that are not allowed: Only (TCP Profile, SCTP Profile, DIAMETER Profile, Diameter Session Profile, Diameter Router Profile, Diameter Endpoint, SIP Profile, SIP Session Profile, SIP Router Profile, DoS Profile, profile statistics)
Conditions:
Create virtual server or transport config which contains both SCTP and Generic Message profile.
Impact:
You are unable to combine the Generic Message profile with the SCTP profile.
Component: Access Policy Manager
Symptoms:
If a client browser opens a WebSocket connection, Portal Access may not send data to the client if the first message in the new connection comes from the server.
Conditions:
- Web application with WebSocket connection
- First data in WebSocket connection is sent from server to client
Impact:
Data is not delivered to the client browser via the WebSocket connection.
Component: Local Traffic Manager
Symptoms:
HTTP/2 request is not forwarded and RST_STREAM with PROTOCOL_ERROR is sent back to the client.
Conditions:
HTTP/2 request with method OPTIONS and pseudo header :path value equal to something other than '*' (asterisk).
Impact:
HTTP/2 request with Method OPTIONS is limited to :path '*' only. Any other URIs are not forwarded to the server but are rejected with RST_STREAM with PROTOCOL_ERROR.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
The BIG-IP system supports QUIC and HTTP/3 draft-24 and draft-25. IETF released draft-27 in February 2020, and major browser vendors have announced they intend to widely deploy support for it, unlike previous drafts.
Conditions:
Browser requests draft-27.
Impact:
Connection downgrades to an older version, or fails if the browser cannot downgrade.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
If a virtual server is configured with both client-side and server-side using HTTP/2, and with translate-address disabled, the connection to the server-side does not succeed.
Conditions:
-- HTTP/2 profiles on both client-side and server-side, using an http-router profile.
-- Translate-address is disabled.
Impact:
Connections fail.
Workaround:
None.
Component: Application Security Manager
Symptoms:
When searching by "Response contains" or "Response does not contain", an incorrect filter is applied and displayed
Conditions:
This occurs in the GUI when selecting "Response contains" or "Response does not contain" filter
Impact:
You are unable to search by response in the GUI
Workaround:
There is no way to search in GUI, but you can search using REST API
Component: TMOS
Symptoms:
The timestamp value of sysUpTime in SNMP traps reported by BGP is incorrect.
Conditions:
BGP connection with a peer flaps, and sends traps for the following:
bgpSnmpNotifyEstablished
bgpSnmpNotifyBackwardTransition
Impact:
The sysUpTime in the trap generated by BGP is incorrect.
Workaround:
None.
Component: TMOS
Symptoms:
If the underlying virtio platform specifies RX and/or TX queue sizes that are 4096 or larger, the BIG-IP system cannot allocate enough contiguous memory space to accommodate this. Errors similar to these are seen in the tmm log files:
ndal Error: Failed to allocate 2232336 (2228224 + 4096 + 16) bytes
virtio[0:7.0]: Error: Failed to allocate descriptor chain
virtio[0:7.0]: Error: Failed allocate indirect rx buffers
Conditions:
-- Using a BIG-IP Virtual Edition (VE) with virtio drivers.
-- The underlying platform specifies RX and/or TX queue sizes of 4096 or larger.
Impact:
TMM continually restarts.
Workaround:
Use the sock driver instead of virtio.
In your BIG-IP VE VM execute the lspci command to determine which virtio driver is present:
# lspci -nn | grep -i eth | grep -i virtio
00:03.0 Ethernet controller [0200]: Red Hat, Inc Virtio network device [1af4:1000]
00:04.0 Ethernet controller [0200]: Red Hat, Inc Virtio network device [1af4:1000]
00:0b.0 Ethernet controller [0200]: Red Hat, Inc Virtio network device [1af4:1000]
Configure a socket driver:
echo "device driver vendor_dev 1af4:1000 sock" > /config/tmm_init.tcl
Reboot the instance
Component: Application Security Manager
Symptoms:
When either DoS Application Profile or Bot Defense profile are used, or ASM Policy with complex LTM Policy, the Accept-Encoding request header is removed by the BIG-IP system, which causes the backend server to respond with uncompressed payload.
Conditions:
One of these options:
-- Bot Defense Profile is associated with the Virtual Server.
-- DoS Profile is associated with the Virtual Server and has Application (L7) enabled.
-- ASM Policy is associated with the Virtual Server and has complex LTM Policy: multiple ASM Policies, or additional rules.
Impact:
-- Response payload sent by the backend server is uncompressed.
-- Performance impact caused by response parsing.
Workaround:
The workaround is to disable the option for modification of Referer header:
tmsh modify sys db asm.inject_referrer_hook value false
Note: Using this brings back the impact of bug 792341.
Component: Application Security Manager
Symptoms:
Under some settings and load, the RPC handler for the botd process restarts frequently, causing unnecessary churn and message-cluttered logs.
Conditions:
-- Bot protection is enabled.
-- A high volume of bot attacks are handled.
Impact:
The RPC handler for the botd process restarts frequently, causing unnecessary churn and noisy logs
Workaround:
None.
Component: Application Security Manager
Symptoms:
When DoSL7 and Bot Defense profiles are configured together on the same Virtual Server, some requests might not be handled by the Bot Defense profile.
Conditions:
-- DoSL7 profile is attached to the virtual server (with Application).
-- Bot Defense profile is attached to the virtual server.
-- Another security module is attached to the virtual server (WebSafe, MobileSafe, ASM).
Impact:
Some requests might not be processed by the Bot Defense profile.
Workaround:
Disable dosl7.idle_fast_path:
tmsh modify sys db dosl7.idle_fast_path value disable
Component: TMOS
Symptoms:
When configuring system auth to use LDAP, if you set check-host-attr to enabled, the resulting /config/bigip/pam.d/ldap/system-auth.conf that is generated cannot be parsed by nslcd.
Conditions:
-- Configuring system auth to use LDAP.
-- Setting check-host-attr to enabled.
Impact:
LDAP-based auth does not function.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
Under certain conditions, TMM may crash while processing iRules LX commands
Conditions:
-iRules LX in use
Impact:
TMM crash, leading to a failover event.
Workaround:
None.
Component: TMOS
Symptoms:
During classification update, you get an error:
"Error: Exception caught in script. Check logs (/var/log/hitless_upgrade.log) for details"
An additional diagnostic is that running the command "/usr/bin/crt_cache_path" reports "none".
Conditions:
This is encountered while updating the classification signatures or the protocol inspection updates.
It can occur when something goes wrong during license activation, but license activation ultimately succeeds.
Impact:
When this issue occurs, auto classification signature update will fail.
Workaround:
You may be able to recover by re-activating the BIG-IP license via tmsh.
Component: Application Security Manager
Symptoms:
After configuring the "validate upon request" option in "Cross Domain Requests" in a Bot Defense profile, JS challenges continue to be sent.
Conditions:
-- Bot Defense profile is enabled
-- "Cross Domain Request":"validate upon request" option is enabled
-- A browser navigates to a qualified (HTML) page from a related domain.
Impact:
Browser receives another JS challenge, instead of retrieving the cookie from the related domain. This causes extra latency for the client.
Workaround:
Use "validate in a bulk" option.
Component: Local Traffic Manager
Symptoms:
If one of the *.info files in /var/rrd becomes corrupted, statsd fails to load it and ends up restarting continuously. You see the following messages in /var/log/ltm:
-- err statsd[766]: 011b020b:3: Error 'Success' scanning buffer '' from file '/var/rrd/throughput.info'.
-- err statsd[766]: 011b0826:3: Cluster collection start error.Exitting.
Conditions:
Corrupted *.info file in /var/rrd.
Impact:
Stats are no longer accurate.
Workaround:
It might take multiple attempts to repair the *.info files. You might have to run the following command several times for different .info files, where <filename> is the actual name of the file (e.g., 'throughput.info'):
found=0;while [ $found != 1 ]; do filetype=`file throughput.info | cut -d " " -f2`;if [[ $filetype != "ASCII" ]]; then rm -f <filename>.info; else grep CRC <filename>.info;found=1;fi; done
Component: Local Traffic Manager
Symptoms:
TMM cores.
Conditions:
-- Platform: B4400 Blade (BIG-IP VPR-B4450N).
-- High availability (HA) mirroring is set up.
-- Provisioned modules: LTM, AFM.
-- HA mirroring messages are received with unexpected interface name length.
Impact:
Processing of invalid length can cause memory corruption. The tmm process generates a core. Traffic disrupted while tmm restarts.
Workaround:
None.
Component: TMOS
Symptoms:
Incorrect value in Transparent Nexthop property on virtual server page with assigned VLAN.
Conditions:
-- Virtual server configured with with transparent next-hop bychecking 'Transparent Nexthop' in the GUI on the LTM Virtual Server page: Transparent Nexthop = None
Works fine with:
Impact:
Incorrect value shown in Transparent Nexthop property field.
Workaround:
Use tmsh to complete the action successfully.
Component: Local Traffic Manager
Symptoms:
In some cases, HTTP requests may fail if system failover occurs immediately after the TLS handshake finishes.
Conditions:
-- System failover to standby device with SSL connection mirroring.
-- Failover occurs immediately after the TLS handshake completes but before the HTTP request.
Impact:
Connection might fail the HTTP request; in some cases, the server may reset HTTP 1.0 requests.
Workaround:
None.
Component: TMOS
Symptoms:
The shutdown sequence of the sflow_agent can include a timeout waiting for a response that results in an assert and core file.
Conditions:
BIG-IP reboot can cause the sflow_agent to core.
Impact:
There is a core file in the /var/core directory after a system reboot.
Component: TMOS
Symptoms:
You are unable to log in to iControl REST via /mgmt/toc/.
Also a NullPointerException is logged to /var/log/restjavad log.
Conditions:
This can be encountered intermittently while using iControl REST.
Impact:
Login failure.
Workaround:
None.
Component: TMOS
Symptoms:
As a part of FPGA firmware update, "tmsh load sys config" fails.
Chmand reports errors:
chmand[19052]: FPGA firmware mismatch - auto update, No Interruption!
chmand[19052]: 012a0006:6: FPGA HSB firmware uploading now...use caution!
Reloading fw_update_post configuration (via systemctl): [FAILED]
Conditions:
Running either of the following commands:
tmsh load sys config
/etc/init.d/fw_update_post reload
Impact:
Firmware update fails.
Workaround:
Use this procedure:
1. Mount /usr:
mount -o rw,remount /usr
2. Add the following line to the '/usr/lib/systemd/system/fw_update_post.service' file:
ExecReload=/etc/init.d/fw_update_post reload
3. Reload systemctl:
systemctl daemon-reload
4. Reload the file:
/etc/init.d/fw_update_post reload
Component: Service Provider
Symptoms:
Tmm crashes with the following log message.
-- crit tmm1[11661]: 01010289:2: Oops @ 0x2a3f440:205: msg->ref > 0.
Conditions:
This can be encountered while passing diameter traffic when one or more of the pool members goes down and retransmissions occur.
Impact:
Traffic disrupted while tmm restarts.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
Tmm crashes. /var/log/tmm contains the log entries:
tmm1: notice panic: invalid route type
tmm1: notice ** SIGFPE **
Conditions:
The issue is intermittent.
1. There is more than one route domain in the parent-child relationship.
2. There are routing entries for the parent route-domain good enough to be selected as an egress point for the routing object (for instance, pool member) which is from child route domain.
3. The routing entry from a parent route domain is selected as an egress point for the object from the child route domain.
4. A new routing entry for child route domain is added.
Impact:
TMM crashes. Traffic disrupted while tmm restarts.
Workaround:
There is no way to workaround a problem, but there is a safe way to add and delete routes without putting a BIG-IP into a state where it could encounter this issue.
Safe way to add/delete a route.
1) Add routes to child route domains first, then to parent route domain.
2) Delete routes from parent route domain first, then from child route domain.
Component: Fraud Protection Services
Symptoms:
JavaScript engine file is empty.
Conditions:
Original HTML page with FPS injected content is cached for too long due to some caching headers (e.g., ETag), so the JavaScript engine link becomes invalid.
Impact:
No FPS protection for that HTML page.
Workaround:
You can use either workaround:
-- Use an iRule to disable caching for protected HTML pages.
-- Set caching time for protected HTML pages to the same value as the datasync tables regeneration timer according to the active datasync profile (default value is two 2 days).
Component: Access Policy Manager
Symptoms:
While trying to create an Access profile, the GUI reports a general database error. There are errors in /var/log/tomcat:
profiles.ProfileUtils$SettingsHandler:error - java.sql.SQLException: Column not found: SOURCE in statement [INSERT into
profile_access
Conditions:
This occurs when you try to create an Access Profile of type SSO from the GUI.
Impact:
You are unable to create the profile using the GUI.
Workaround:
You can create the Access Profile using TMSH.
tmsh create access access_test_sso type sso accept-languages add { en } sso-name sso_test1
Component: Service Provider
Symptoms:
DIAMETER router messages queued for retransmission may not be deleted until the connection closes.
Conditions:
-- Diameter transmission setting is enabled and a DIAMETER message is queued for retransmission.
-- The retransmission for the message is not triggered
Impact:
The memory used to hold the copy of the message in the retransmission queue is leaked.
Workaround:
None.
Component: Application Security Manager
Symptoms:
A log that has 'Browser Automation’ as the ‘Bot Category’ exists.
When filtering for only Bot Category: Browser Automation, nothing Shows up.
Conditions:
-- ASM provisioned.
-- Filtering by Bot Categories on Bot Requests Log
Impact:
Legitimate requests being blocked but cannot filter on the category to narrow down their focus.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
Clients attempting to connect with QUIC version 25 and ALPN h3-25 are unable to connect.
Conditions:
An end user client attempts to connect using QUIC version 25 and ALPN h3-25.
Impact:
Attempts to use HTTP/3 with some clients may fail.
Workaround:
None.
Component: Advanced Firewall Manager
Symptoms:
Due to a large number of tables containing 'Protected Object Aggregate stats', the merged daemon might cause elevated CPU usage on odd-numbered CPU cores.
Conditions:
AFM, ASM, or DoS features are provisioned.
Impact:
Elevated CPU usage on odd-numbered cores caused by merged daemon.
Workaround:
None.
Component: Advanced Firewall Manager
Symptoms:
Value in mitigation thresholds are above infinite value (4294967295)
Conditions:
Multiplier based mitigation mode for Dos static vectors after run relearn thresholds command
Impact:
Incorrect display value for DoS thresholds in GUI and tmctl
Component: Advanced Firewall Manager
Symptoms:
There are no hardware mitigation for DoS attacks
Conditions:
Auto-threshold mode and detection for attacked destinations should be activate for DoS static vector
Impact:
DoS attack mitigation performed only by software
Component: Application Security Manager
Symptoms:
When using CSRF protection, sensitive parameters values can be masked incorrectly in the event log.
Conditions:
The request contains a CSRF token and sensitive parameters.
Impact:
Sensitive parameters values can be masked incorrectly in the event log.
Workaround:
None.
Component: Access Policy Manager
Symptoms:
APM virtual server user's GUI (e.g., 'Logon page') cannot be rendered by browsers.
Conditions:
This issue is encountered when an XML profile is configured for the APM virtual server.
Impact:
APM end users are unable to get a logon page.
Workaround:
Disable the XML profile for the APM virtual server.
Component: TMOS
Symptoms:
Mcpd restarts and produces a core file.
Conditions:
This can occur while adding a pool member to a NAPTR GTM pool where the flag used is an uppercase 'A' or 'S' character.
Impact:
Mcpd crash and restart results in high availability (HA) failover.
Workaround:
Use a lowercase 'a' or 's' as the flag value.
Component: Access Policy Manager
Symptoms:
Per-Request Policy (PRP) Access Policy with Customization Type set to Modern cannot be created due to internal error.
Conditions:
Creating a PRP Access Policy with Customization Type set to Modern.
Impact:
Administrator cannot use modern customization.
Workaround:
1. In bigip.conf find the following line:
apm policy customization-source /Common/standard { }
2. Add the following line:
apm policy customization-source /Common/modern { }
3. Save the changes.
4. Load the config:
tmsh load sys config
Component: TMOS
Symptoms:
Due to a known issue, software upgrade to an engineering hotfix might fail with a log message in /var/log/ltm similar to:
info lind[5500]: 013c0007:6: Install complete for volume=HD1.2: status=failed (Software compatibility tests failed.)
Conditions:
Performing a software upgrade to a hotfix release on a vCMP guest.
Impact:
Unable to perform an upgrade.
Workaround:
Option 1:
Make sure that .iso files for both base image and hotfix reside only on a vCMP guest before starting the installation.
Option 2:
Even if the hotfix installation has failed, the base image should still have been installed properly, so you can restart the vCMP guest and perform a hotfix installation on top of already installed base image.
Option 3:
Even if the hotfix installation has failed, the base image should still have been installed properly. Ensure there is copy of the hotfix locally within the vcmp Guest. Then restart the lind service
tmsh restart sys service lind
The hotfix installation should begin again this time using the hotfix from within the guest /shared/images/ location.
Component: Application Visibility and Reporting
Symptoms:
TMM CPU utilization around 100 percent under Security ›› Reporting : ASM Resources : CPU Utilization.
Conditions:
No special conditions. Only viewing at the stats of TMM CPU in 'Security ›› Reporting : ASM Resources : CPU Utilization'. They will always be in wrong scale, but when the TMM has ~1% CPU usage, it will be presented as 100% CPU usage.
Impact:
Wrong scale is presented and might cause machine's state to be interpreted wrongly.
Workaround:
1. Backup /etc/avr/monpd/monp_asm_cpu_info_measures.cfg file.
2. Run the following:
$ sed -i 's|tmm_avg_cpu_util)/(count(distinct time_stamp)|tmm_avg_cpu_util)/(count(distinct time_stamp)*100|g' /etc/avr/monpd/monp_asm_cpu_info_measures.cfg
3. Compare the backup file to /etc/avr/monpd/monp_asm_cpu_info_measures.cfg:
Make sure that there are two lines modified, and that the modification is multiplying with 100 the denominator (i.e., actually dividing the TMM value with 100).
4. To make those changes take affect, run the following command:
$ bigstart restart monpd
Component: TMOS
Symptoms:
The /var logical volume size of 950 MiB for LTM-only BIG-IP Virtual Edition (VE) images may be too small for some deployments. This can result in result in loss of SSH access.
Conditions:
This applies to deployments that use declarative onboarding for configuration.
Impact:
Complex declarative onboarding configurations may fill the /var logical volume. You are locked out because of the too-small volume.
Workaround:
The workaround is to manually extend the /var logical volume.
For more information, see K14952: Extending disk space on BIG-IP VE :: https://support.f5.com/csp/article/K14952.
Behavior Change:
The size of the /var logical volume was increased from 950MiB to 1.5GiB for LTM-only Virtual Edition images.
Component: Global Traffic Manager (DNS)
Symptoms:
Load balance methods for Link Controller inbound wide IP are always set to default values when the load balancing method is updated through GUI.
Conditions:
-- Multiple inbound wide IPs are configured;
-- Load balancing methods are updated through GUI once.
Impact:
Unable to manage wide IPs through the GUI.
Workaround:
Use tmsh to manage Inbound WideIPs.
Component: Application Security Manager
Symptoms:
Long request buffers are kept around for a long time in bd.
Conditions:
-- The answer_100_continue internal parameter is turned off (non default) or the version is pre 15.1
-- The server closes the connection while request packets are accumulated.
Impact:
The long request buffers are consumed. You may see a "Too many concurrent long requests" log message and requests with large content lengths will get reset.
Workaround:
There is no workaround that can be done from ASM configuration.
If possible, change the server application settings to wait longer for the request payload in 100-continue request or change the client side application to not work with 100-continue.
Component: Application Security Manager
Symptoms:
When using an iRule to disable bot profile, and causing it to be disabled (for the entire connection) during a CAPTCHA challenge -- TMM will crash.
Conditions:
-- Bot Defense profile is attached to the Virtual Server, with a CAPTCHA mitigation.
-- An iRule is attached to the virtual server, which disables bot profile.
-- Sending a request that is responded with a CAPTCHA, then sending (in the same connection), a request that disable the bot profile, and then answering the CAPTCHA.
Impact:
Traffic disrupted while tmm restarts.
Workaround:
When using an iRule to disable bot defense profile on certain conditions, add an "else" clause for re-enabling the profile, taking note that all ::disable iRule commands are effective for the entire connection, and not just the transaction.
Component: Service Provider
Symptoms:
DIAMETER messages queued for retransmission that do not receive answer responses may be missed by the sweeper logic and not be deleted until the connection closes.
Conditions:
-- A DIAMETER message is queued for retransmission without a timeout to tigger retransmission.
-- No answer response is received.
Impact:
The memory used to hold the copy of the message in the retransmission queue is leaked.
Workaround:
None.
Component: TMOS
Symptoms:
If a VLAN tag is configured on the Virtual Function in the host, and the BIG-IP guest is configured to use a tagged VLAN, packets that egress the host on this VLAN contain only the VLAN tag configured on the host (i.e. the BIG-IP's VLAN tag is lost).
Conditions:
- Using a BIG-IP VE.
- A VLAN tag is configured on both the host VF and on the BIG-IP.
Impact:
The BIG-IP's VLAN tag is lost.
Component: TMOS
Symptoms:
You are unable to associate new ASM policies to LTM policies, due to REST timeout.
Conditions:
This can be encountered when there are a large number of policies configured in ASM.
Impact:
Unable to associate new ASM policies to LTM policies, due to rest timeout.
Workaround:
None.
Component: TMOS
Symptoms:
The memory pages available and in use are tracked with system statistics. Previously those statistics were available only with the tmctl command in the shell.
Conditions:
When system resource decisions are being made, the information about memory usage is important.
Impact:
It is not feasible to query each BIG-IP device separately.
Workaround:
None.
Component: Access Policy Manager
Symptoms:
HTTP Strict-Transport-Security (HSTS) headers have an empty value for some APM Access Policy-generated responses.
Conditions:
This occurs when the following conditions are met:
-- HTTP profile is configured with HSTS mode=disabled (which it is by default).
-- HTTP requests for APM renderer content, including CSS, JS, and image files from the webtop.
Impact:
Some audit scanners can consider the empty value of Strict-Transport-Security headers as a vulnerability. For browsers, the empty HSTS value equals no HSTS in response.
Workaround:
1. Enable HSTS mode for the HTTP profile.
2. Use an iRule to remove the empty HSTS header from responses:
when HTTP_RESPONSE_RELEASE {
if { [HTTP::header value "Strict-Transport-Security"] eq "" } {
HTTP::header remove "Strict-Transport-Security"
When the HTTP profile is configured with HSTS mode=disabled, responses from APM renderer content are now sent without an HSTS header.
Component: Application Visibility and Reporting
Symptoms:
The MaxMemory Attribute is not reported in the System Monitor statistics report.
Conditions:
This is encountered when viewing the System Monitor report.
Impact:
No 'MaxMemory' value label appears in System Monitor statistics. Instead, there are duplicate AvgMemory fields, for example:
...(AvgMemory='3818',AvgMemory='3818').
Workaround:
Use the AvgMemory value that is the higher of the two to represent MaxMemory.
Note: Sometimes, the AvgMemory and MaxMemory values are the same. In that case, use the second value.
Component: Local Traffic Manager
Symptoms:
TMM may sometimes core when HTTP-MR proxy attempts to go into passthrough mode.
Conditions:
-- HTTP profile is attached to the virtual server.
-- The httprouter profile is attached to the virtual server.
-- HTTP goes into passthrough mode for any variety of reasons.
Impact:
Traffic disrupted while tmm restarts.
Workaround:
None.
Component: Access Policy Manager
Symptoms:
If the security service attempts server connection reuse, client port reuse causes RST on new connections.
Conditions:
-- Service profile is attached to virtual server.
-- SSL Orchestrator (SSLO) is licensed and provisioned and Service chain is added in the security policy.
-- Security service reuses server-side connection.
-- Client reuses the source port.
Impact:
The BIG-IP system or SSLO rejects new connection from clients when a client reuses the port.
Workaround:
None.
Component: Service Provider
Symptoms:
In MRF/Diameter deployment, mirrored connections on the standby may be lost when the "process ingress error" log is observed only on the standby, and there is no matching log on the active.
Conditions:
This can happen when there is a large amount of mirror traffic, this includes the traffic processed by the active that requires mirroring and the high availability (HA) context synchronization such as persistence information, message state, etc.
Impact:
Diameter mirror connections are lost on the standby. When failover occurs, these connections may need to reconnect.
Component: Application Security Manager
Symptoms:
BD crash on specific scenario
Conditions:
A brute force attack mitigation using captcha or client side challenge.
Impact:
BD crash, failover.
Workaround:
Add an iRule that removes the query string from the referrer header only for the login page POSTs.
Component: TMOS
Symptoms:
When BGP tries to print the address of the default route's peer but there is no matching address for IPv4 or IPv6 so the system returns a NULL and attempting to print results in a crash.
Conditions:
-- Running the show command: sh bgp ipv6 ::/0.
-- There is no matching IPv4 or IPv6 address for the peer.
The conditions that cause this to occur are unknown.
Impact:
Bgdp crashes. Routing may be affected while bgpd restarts.
Workaround:
None.
Component: TMOS
Symptoms:
The tuning values programmed in the switch are not correct for Finisar OPT-0039 QSFP28 modules.
Conditions:
-- Using Finisar OPT-0039 QSFP28 modules.
-- Running on i15000 and i15800 platforms.
Note: Use 'tmsh list net interface vendor-partnum', to identify the optic modules installed.
Impact:
You might see traffic drop.
Note: Potential issues related to incorrect tuning values come from F5-internal sources and have not been reported in production configurations.
Workaround:
None.
Component: Application Visibility and Reporting
Symptoms:
AVRD cores during startup.
Conditions:
Avrd receives a SIGTERM while it is starting.
Impact:
This can lead to an AVRD core.
Component: TMOS
Symptoms:
The ifLinkUpDownTrapEnable setting is not saved to config files nor UCS. If you have disabled 'link up/down' traps for an interface, save the config, and then load the config files or UCS, all interfaces will have traps enabled, even the ones that were explicitly disabled.
Conditions:
-- Disable 'link up/down' traps for an interface.
-- Save the configuration or UCS.
-- Reload the configuration or load the UCS.
Impact:
All interfaces have traps enabled, even the ones that were explicitly disabled.
Workaround:
None.
Component: TMOS
Symptoms:
ASM policies may not load immediately after upgrade due to SELinux policies issues relating to the upgrade process.
Conditions:
1. ASM is provisioned.
2. One or more ASM Security Policies is attached to one or more virtual servers.
3. Upgrade from v12.x or v13.x to v14.x or later.
Impact:
Traffic is not processed properly after upgrade due to failure to load ASM policies.
Workaround:
You can use either of the following workarounds.
-- Remove ASM Policies while upgrading:
1. Prior to upgrade, remove all ASM Security Policies from all virtual servers.
2. Upgrade.
3. Reassociate each ASM Security Policy with its original virtual server.
-- Restore the UCS on a new boot location after upgrade:
1. Prior to upgrade, create a UCS.
2. Upgrade or create a new instance of the software version at the target location.
3. Restore the UCS at the new location.
Component: Application Security Manager
Symptoms:
After changing a parent policy's learning mode or other learning attributes in policy-builder settings, deploying the policy will result in differences in the child policies.
Conditions:
On BIG-IP and BIG-IQ:
1. Parent policy has a policy-building section that is inherited.
2. Child policy has wildcard default (*) elements such as urls.
On BIG-IQ:
3. Change parent learning mode from manual to disabled or vice versa
4. Deploy changes
Impact:
There are differences after deploy.
Workaround:
Discover and deploy again from BIG-IQ
Component: Application Visibility and Reporting
Symptoms:
The scheduled report email is sent from BIG-IP using TLS even if configured to not use encryption. When the mail server TLS is outdated it may lead to failure of the mail delivery.
Conditions:
The scheduled reports are enabled and configured to use a mail server which reports TLS capability.
Impact:
The minor impact is unexpected behaviour. In rare cases it may lead to malfunction of the scheduled reports.
Component: Application Visibility and Reporting
Symptoms:
AVR report mailer times out prematurely and reports errors:
AVRExpMail|ERROR|2019-11-26 21:01:08 ECT|avrmail.php:325| PHPMailer exception while trying to send the report: SMTP Error: data not accepted.
Conditions:
Configure reports, which will be sent to e-mail
Impact:
Error response from SMTP server, and the report is not sent
Workaround:
Increase timeout in avrmail.php via bash commands
Component: Local Traffic Manager
Symptoms:
- MPTCP enabled TCP connection is in SYN_RECEIVED state.
- TMM cores.
Conditions:
- MPTCP is enabled.
- SYN/ACK (with MP_JOIN or MP_CAPABLE) sent by the BIG-IP is not ACKed and needs to be retransmitted.
Impact:
Traffic disrupted while tmm restarts.
Workaround:
Disable MPTCP option in the TCP profile.
Component: Local Traffic Manager
Symptoms:
After configuring a clientssl-quic profile, you get a validation error:
01b40001:3: A cipher group must be configured when TLS 1.3 is enabled (validation failed for profile /Common/clientssl-f5quic-udp).
Conditions:
This can occur when using the clientssl-quic built-in profile to build a profile that can serve HTTP/3 over QUIC.
Impact:
You are unable to configure a clientssl profile to work with HTTP/3 + QUIC that is also customized to serve the right certificate, etc.
Workaround:
Modify the clientssl-quic profile to have the following properties:
cipher-group quic
ciphers none
This requires the following additional config objects:
ltm cipher group quic {
allow {
quic { }
ltm cipher rule quic {
cipher TLS13-AES128-GCM-SHA256,TLS13-AES256-GCM-SHA384
description "Ciphers usable by QUIC"
Update the built-in configuration to pass validation.
Component: Local Traffic Manager
Symptoms:
TMM crashes while handling HTTP response
Conditions:
HTTP virtual server performing decompression of response data from a server, e.g. because a rewrite profile is attached to the virtual server.
Impact:
Traffic disrupted while tmm restarts.
Workaround:
Disable compression on the server.
Component: TMOS
Symptoms:
MCPD can crash with out of memory when there are many bigd processes (systems with many CPU cores) and many pool members/nodes/monitors.
As a guideline, approximately 100,000 pool members, nodes, and monitors can crash a system that has 10 bigd processes (BIG-IP i11800 platforms). tmm crash
Conditions:
-- Tens of thousands of pool members, nodes, and/or monitors.
-- Multiple (generally 6 or more) bigd processes.
-- System startup or bigstart restart.
Impact:
The mcpd process crashes. Traffic disrupted while mcpd restarts.
Workaround:
Set the db variable bigd.numprocs to a number smaller than the number of bigd processes currently being started.
Component: TMOS
Symptoms:
High CPU usage by obfuscator for an extended period of time.
Conditions:
Occurs very rarely, when FPS or L7 DDoS protection are enabled.
Impact:
High CPU Usage.
Workaround:
Kill the obfuscator process
Component: Local Traffic Manager
Symptoms:
Under rare timing conditions, one or more ephemeral nodes and pool members may be created for the wrong FQDN name, resulting in one or more ephemeral pool members being created incorrectly for a given pool.
Conditions:
This problem occurs when a DNS Request is sent to resolve a particular FQDN name with the same DNS Transaction ID (TXID) as another DNS Request currently pending with the same DNS name server. When this occurs, the IP addresses returned in the first DNS Response received with that TXID may be incorrectly associated with a pending DNS Request with the same TXID, but for a different FQDN name which does not actually resolve to those IP addresses.
The timing conditions that produce such duplicate TXIDs may be produced by one or more of the following factors:
1. Many FQDN names to be resolved.
2. Short DNS query interval values configured for the FQDN template nodes (or short TTL values returned by the DNS name server with the query interval configured as 'ttl').
3. Delayed responses from the DNS name server causing DNS queries to remain pending for several seconds.
Impact:
When this issue occurs, traffic may be load-balanced to the wrong members for a given pool.
Workaround:
It may be possible to mitigate this issue by one or more of the following actions:
-- Ensuring that the DNS servers used to resolve FQDN node names have sufficient resources to respond quickly to DNS requests.
-- Reducing the number of FQDN template nodes (FQDN names to be resolved).
-- Reducing the frequency of DNS queries to resolve FQDN node names (FQDN names) by either increasing the 'interval' value configured for FQDN template nodes, or by increasing the TTL values for DNS zone records for FQDN names for FQDN nodes configured with an 'interval' value of 'ttl'.
Component: Service Provider
Symptoms:
In iRules, when "GENERICMESSAGE::message create" is called after "reject" command inside "after -periodic", it may cause core. Below is an example iRules.
when CLIENT_ACCEPTED {
... omitted ...
after 1000 -periodic {
... omitted ...
reject
GENERICMESSAGE::message create "test"
This relates to ID 859113.
Conditions:
GENERICMESSAGE::message create" is called after "reject" inside "after -periodic
Impact:
Traffic disrupted while tmm restarts.
Workaround:
There are 2 possible work-arounds
- use "return" command after "reject" to exit after script immediately after "reject" command is invoked
- add routine to cancel the after in CLIENT_CLOSED event
Component: Local Traffic Manager
Symptoms:
In iRules, when "reject" is used inside "after -periodic" and it is followed by "GENERICMESSAGE::message create". It may trigger a tmm core. Below is an example iRule.
when CLIENT_ACCEPTED {
... omitted ...
after 1000 -periodic {
... omitted ...
reject
GENERICMESSAGE::message create "test"
This relates to ID 859721
Conditions:
- "reject" is used inside "after -periodic"
- it is followed by "GENERICMESSAGE::message create"
Impact:
Traffic disrupted while tmm restarts.
Workaround:
There are 2 possible work-arounds
- use "return" command after "reject" to exit after script immediately after "reject" command is invoked
- add routine to cancel the after in CLIENT_CLOSED event
Component: TMOS
Symptoms:
The net-snmp 5.7.2 library does not support extended key lengths for SHA and AES protocols used for SNMPv3 authentication and privacy protocols.
Conditions:
When the BIG-IP net-snmp libraries are version 5.7.2, or earlier, than only SHA and AES are available for configuring trap sessions and users in SNMPv3.
Impact:
The longer keys lengths for SNMPv3 cannot be used.
Component: Local Traffic Manager
Symptoms:
ICMP packets are forwarded to both virtual wire interface, which causes MAC-Flip on the connected switches.
Conditions:
-- Ingress ICMP packet is on one TMM.
-- Egress is on another TMM.
Impact:
Traffic is disrupted in the network.
Workaround:
None.
Component: Application Security Manager
Symptoms:
XML with sensitive data gets to the ICAP server, even when the XML profile is not configured to be inspected.
Conditions:
XML profile is configured with sensitive elements on a policy.
ICAP server is configured to inspect file uploads on that policy.
Impact:
Sensitive data will reach the ICAP server.
Workaround:
No immediate workaround except policy related changes
Behavior Change:
An internal parameter has been added, called send_xml_sensitive_entities_to_icap, and the default value is 1.
When this is changed to 0 (using this command):
/usr/share/ts/bin/add_del_internal add send_xml_sensitive_entities_to_icap 0
XML requests with sensitive data will not be sent to ICAP.
Component: TMOS
Symptoms:
Merged crashes when system memory is exhausted
Conditions:
System memory is is at 0% available.
Impact:
Merged crashes, stopping stats updates
Workaround:
Reduce the configuration on the system
Component: Device Management
Symptoms:
When a large number of LTM objects are configured on BIG-IP, making updates via iControl REST can result in restjavad/restnoded/icrd errors.
Conditions:
Using iControl REST/iapp to update a data-group that contains a large number of records, e.g., 75,000 or more.
Impact:
REST operations can time out when they take too long, and it is not possible to increase the timeout.
Workaround:
None.
Behavior Change:
New Sys DB variables have been added to allow you to modify the timeout settings of restjavad, restnoded, and icrd:
restnoded.timeout
restjavad.timeout
icrd.timeout
The default value is 60 seconds for each of these.
A restart of restjavad and restnoded is required for the change to take effect.
tmsh restart /sys service restjavad
tmsh restart /sys service restnoded
Component: Application Security Manager
Symptoms:
Under certain conditions, Proactive Bot Defense may redirect clients to an unvalidated path.
Conditions:
-Proactive Bot Defense enabled.
Impact:
Clients may be redirected to an unvalidated path.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
Whenever the server or client side data have not been drained, 'server drained' or 'client drained' appear in /var/log/tmm as errors.
Conditions:
-- Using iRule configuration with LB::detach or LB::connect.
-- Server- or client-side data has not been drained before those statements are triggered.
Impact:
TMM crashes and can cause an outage on standalone system or failover in a DSC. Traffic disrupted while tmm restarts.
Workaround:
None.
Component: Access Policy Manager
Symptoms:
On the Citrix Workspace app, clicking 'Refresh Apps' after signing out fails with message "Refresh Failed" with v15.1.x
Conditions:
-- Running the Citrix Workspace all.
-- Clicking 'Refresh Apps' after signing out.
-- Running software v15.1.x.
Impact:
The system reports a 'Refresh failed' error, and the app must to be reset.
Workaround:
None.
Component: TMOS
Symptoms:
Despite having free memory, the BIG-IP system frequently logs kernel page allocation failures to the /var/log/kern.log file. The first line of the output appears similar to the following example:
swapper/16: page allocation failure: order:2, mode:0x104020
After that, a stack trace follows. Note that the process name in the line ('swapper/16', in this example) varies. You may see generic Linux processes or processes specific to F5 in that line.
Conditions:
This issue is known to occur on the following VIPRION blade models:
- B2250 (A112)
- B4300 (A108)
- B4340N (A110)
- B4450 (A114)
Please note the issue is known to occur regardless of whether or not the system is running in vCMP mode, and regardless of whether the system is Active or Standby.
Impact:
As different processes can experience this issue, the system may behave unpredictably. For example, it is possible for a TMOS installation to fail as a result of this issue. Other processes may not exhibit any side effect as a result of this issue. The exact impact depends on which process becomes affected and how this process is designed to handle such a failure to allocate memory.
Workaround:
You can work around this issue by increasing the value of the min_free_kbytes kernel parameter. This controls the amount of memory that is kept free for use by special reserves.
It is recommend to increase this as follows:
-- 64 MB (65536 KB for 2250 blades)
-- 48 MB (49152 KB for B4300 blades)
-- 128 MB (131072 KB for 4450 blades)
You must do this on each blade installed in the system.
When instantiating this workaround, you must consider whether you want the workaround to survive only reboots, or to survive reboots, upgrades, RMAs, etc. This is an important consideration to make, as you should stop using this workaround when this issue is fixed in a future version of BIG-IP software. So consider the pros and cons of each approach before choosing one.
-- If you want the workaround to survive reboots only, perform the following procedure:
1) Log on to the advanced shell (BASH) of the primary blade of the affected VIPRION system.
2) Run the following commands (with the desired amount in KB):
# clsh "sysctl -w vm.min_free_kbytes=131072"
# clsh "echo -e '\n# Workaround for ID753650' >> /etc/sysctl.conf"
# clsh "echo 'vm.min_free_kbytes = 131072' >> /etc/sysctl.conf"
-- If you want the workaround to survive reboots, upgrades, RMAs, etc., perform the following procedure:
1) Log on to the advanced shell (BASH) of the primary blade of the affected VIPRION system.
2) Run the following commands (with the desired amount in KB):
# clsh "sysctl -w vm.min_free_kbytes=131072"
# echo -e '\n# Workaround for ID753650' >> /config/startup
# echo 'sysctl -w vm.min_free_kbytes=131072' >> /config/startup
Note that the last two commands are not wrapped inside 'clsh' because the /config/startup file is already automatically synchronized across all blades.
Once the issue is fixed in a future BIG-IP version, remove the workarounds:
-- To remove the first workaround:
1) Edit the /etc/sysctl.conf file on all blades, and remove the added lines at the bottom.
2) Reboot the system by running 'clsh reboot'. This will restore the min_free_kbytes kernel parameter to its default value for the BIG-IP version you are running.
-- To remove the second workaround:
1) Edit the /config/startup file on the primary blade only, and remove the extra lines at the bottom.
2) Reboot the system by running 'clsh reboot'. This restores the min_free_kbytes kernel parameter to its default value for the BIG-IP version you are running.
To verify the workaround is in place, run the following command (this should return the desired amount in KB):
# clsh "cat /proc/sys/vm/min_free_kbytes"
Component: Application Security Manager
Symptoms:
Whenever a pool IP address is modified, an update is sent to bd regardless of whether that pool is relevant to ASM. When these updates occur frequently, as can be the case for FQDN nodes that honor DNS TTL, latency can be introduced in ASM handling.
Conditions:
Pool nodes have frequent IP address updates, typically due to an FQDN node set to honor DNS TTL.
Impact:
Latency is introduced to ASM handling.
Workaround:
Set the fast changing nodes to static updates every hour.
Component: Local Traffic Manager
Symptoms:
A TCP connection hangs occasionally.
Conditions:
-- The TCP connection is on the clientside.
-- sys db tm.tcpsendrandomtimestamp is enabled (default is disabled).
-- A virtual server's TCP's Verified Accept and Timestamps are both enabled.
Impact:
TCP connections hangs, and data transfer cannot be completed.
Workaround:
You can use either of the following workarounds:
-- Disable tm.tcpsendrandomtimestamp.
-- Disable either the TCP's Verified Accept or Timestamps option.
Component: Service Provider
Symptoms:
For each message dropped during GENERICMESSAGE_INGRESS, memory is leaked.
Conditions:
Usage of GENERICMESSAGE::message drop iRule command during GENERICMESSAGE_INGRESS event will leak memory.
Impact:
As more memory is leaked, less memory is available for message processing, eventually leading to a core.
Workaround:
Use MR::message drop during MR_INGRESS event instead to drop a message.
Component: Access Policy Manager
Symptoms:
When a form is received in the response, TMM crashes when SSO identifies the form parameter, and logs the Form parameter value and type in SSOv2 form-based passthrough log.
Conditions:
-- When any of the form parameters that SSO receives in the response does not have a value.
-- Passthrough mode is enabled in SSO.
Impact:
TMM crash when Passthrough mode is enabled in SSO. Traffic disrupted while tmm restarts.
Workaround:
Do not use Passthrough mode with SSO.
Component: Local Traffic Manager
Symptoms:
Because the BIG-IP system does not recognize proprietary multicast MAC addresses such as PVST+ (01:00:0c:cc:cc:cd) and STP (01:80:c2:00:00:00), when STP is disabled the system does not drop those frames. Instead the system treats those as L2 multicast frames and forwards between 2 interfaces.
Conditions:
-- STP disabled
-- All platforms except 2000 series, 4000 series, i2000 series, i4000 series and i850.
Impact:
PVST+ (01:00:0c:cc:cc:cd), a proprietary multicast MAC is forwarded instead of discarded, even when STP is disabled.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
TMM cores intermittently when HTTP/3 tries to use uni-directional streams in 0-RTT scenario.
Conditions:
-- Virtual server with QUIC, HTTP/3, HTTP, SSL and httprouter profiles.
-- 0-RTT connection resumption in progress.
Impact:
TMM cores intermittently.
Workaround:
No workaround.
Component: Application Security Manager
Symptoms:
Directory cleanup for for failed Attack Signature Updates (ASU) is too aggressive and may delete needed files in the middle of installation itself, which causes the update to fail.
Conditions:
An ASU runs at the same time as the file cleanup task.
Impact:
The ASU fails to complete successfully.
Workaround:
The default clean interval is 300 seconds (5 minutes).
1. Run the following command to monitor the clean activity:
#tailf /var/log/ts/asmcrond.log | grep CleanFiles
2. Watch for following message in the log:
asmcrond|INFO|Mar 20 21:54:44.389|24036|F5::PeriodicTask::Base::run,,Running Task: CleanFiles
3. Upgrade the ASU immediately.
If 5 minutes is not enough, you can increase the clean interval.
1. Adjust the interval in the /etc/ts/tools/asmcrond.cfg file:
From:
[CleanFiles]
Interval=300
[CleanFiles]
Interval=3000
Important: Do not set Interval too high. 50 minutes (3000 seconds) should be enough.
2. Restart the asmcrond by killing the process. It respawns after several seconds.
ps -ef | grep asmcrond
kill <pid>
3. Monitor the asmcrond.log until you see another Cleanfiles log message.
# tailf /var/log/ts/asmcrond.log | grep CleanFiles
4. Install the ASU; the temp files can stay in the folder for 50 minutes.
5. After the ASU is installed, change the interval back to 300 and restart asmcrond.
6. Make sure asmcrond has been started correctly.
# ps -ef | grep asmcrond
# tailf /var/log/ts/asmcrond.log
Component: Local Traffic Manager
Symptoms:
HTTP/2 connection breaks and Tcl error is logged in /var/log/ltm similar to the following:
TCL error: /Common/http2_disable <CLIENT_ACCEPTED> - Unknown error (line 1) (line 1) invoked from within "HTTP2::disable".
Conditions:
Any of the following Tcl commands are used in any iRule event: HTTP2::enable, HTTP2::enable clientside, HTTP2::disable, HTTP2::disable clientside.
Impact:
HTTP/2 traffic is not passed to the serverside.
Workaround:
Do not use the following Tcl commands: HTTP2::enable, HTTP2::enable clientside, HTTP2::disable, HTTP2::disable clientside
Component: Access Policy Manager
Symptoms:
VMware Horizon clients cannot ,connect to APM and /var/log/apm contains hte following error:
... err tmm3[12345]: 01490514:3: (null):Common:00000000: Access encountered error: ERR_BOUNDS. File: ../modules/hudfilter/access/access.c, Function: access_do_internal_retry, Line: 16431
Conditions:
-- Access Policy has 'VMware View Logon Page' agent configured with 'Kerberos Authentication'.
-- The policy has been in use for some time.
Impact:
VMware Horizon client cannot connect to APM after some time.
Workaround:
None.
Component: Advanced Firewall Manager
Symptoms:
DNS over TCP packet is not rate-limited accurately by DoS device sweep and flood vector.
Conditions:
-- Setting the correct DNS pkt type in the DoS device sweep or flood vector.
-- Sending DNS over TCP.
Impact:
DNS over TCP is DDoS attack is not mitigated correctly.
Workaround:
Using DNS DoS vector to mitigate the attack.
Component: Global Traffic Manager (DNS)
Symptoms:
Big3d fails external monitor SIP_monitor because GTM SIP Monitors need to be running as privileged.
Conditions:
TLS SIP monitor on pool member requiring client auth.
Impact:
Big3d fails external monitor SIP_monitor.
Workaround:
The only workaround is to allow world reading of key files in the filestore, however, this is not ideal as it exposes potentially sensitive data.
Component: TMOS
Symptoms:
When using more than 4 BIG-IP devices connected in a device cluster, and adding 2 more devices to the trust domain, the mcpd processes of each device may get into a sync loop. This causes mcpd to reach up to 90% CPU utilization during this time, and causes other control-plane functionality to halt. This state may last 10-20 minutes in some cases, or continuous in other cases.
Conditions:
-- More than 4 BIG-IP devices are configured in a trust domain configuration.
-- Adding at least 2 more devices to the trust domain, one after the other, without waiting for the full sync to complete.
-- ASM, FPS, or DHD (DOS) is provisioned.
Impact:
High CPU utilization, GUI, TMSH, and REST API not responding or slow-responding, other system processes halted.
Workaround:
When adding a BIG-IP device to the trust domain, before adding any other device, wait a few minutes until the sync is complete, and no more sync logs display in /var/log/ltm.
Component: Local Traffic Manager
Symptoms:
If a 100 Continue response from a server arrives in mulitple packets, HTTP Parsing may not work as expected. The later server response payload may not be sent to the client.
Conditions:
The server responds with a 100 Continue response which has been broken into more than one packet.
Impact:
The response is not delivered to the client. Browsers may retry the request.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
Bigd reporting 'overload' or 'overloaded' in /var/log/ltm.
SSL monitors flapping while the servers are available.
Conditions:
-- FIPS-enabled platform.
-- HTTPS monitors using client-cert authentication where the key is stored in FIPS HSM.
-- Large number of monitors or low interval.
Impact:
Periodic service interruption depending on which monitors are flapping. Reduced number of available servers.
Workaround:
-- Increase the interval on the monitors.
-- Switch the monitors to use software keys.
Component: Advanced Firewall Manager
Symptoms:
Observed autodosd CPU burst
Conditions:
Enable autodosd and a large number of virtual servers
Impact:
High cpu utilization in autodosd
Workaround:
Disable autodosd
Component: Local Traffic Manager
Symptoms:
TMM crash with 'server drained' panic string.
Conditions:
-- Server-side flow is detached while the proxy is still buffering data for the pool member and the client continues to send data.
-- The detach may be triggered by the LB::detach iRule commands or internally.
Impact:
TMM crash, failover, brief traffic outage. Traffic disrupted while tmm restarts.
Workaround:
-- In cases in which the detach is triggered internally, there is no workaround.
-- In cases in which the detach is triggered by LB::Detach, make sure the command is not executed when a request may still be in progress by using it in response events, for example HTTP_RESPONSE, USER_RESPONSE, etc.
Component: Local Traffic Manager
Symptoms:
Memory allocation failures during proxy initialization are ignored. TMM cores when trying to access uninitialized memory.
Conditions:
-- HTTP or HTTP/2 virtual server with httprouter profile.
-- Low memory or fragmented memory on the system when configuration is being loaded.
Impact:
TMM cores when accessing uninitialized memory.
Workaround:
No workaround.
Component: Local Traffic Manager
Symptoms:
QUIC profile has a field for maximum uni-streams. This represents the number of concurrent uni-streams that the peer can create. If HTTP/3 is also configured on the virtual, then the value for uni-streams should ne >=3. The peer should be able to create at least 3 uni-streams, for control, encoder and decoder.
Conditions:
QUIC, HTTP/3, SSL and httprouter profiles are configured on the virtual. QUIC client tries to establish a connection with Big-IP. HTTP/3 is negotiated in ALPN.
Impact:
If fewer than 3 max uni-streams are configured, HTTP/3 transactions will not be successful.
Workaround:
Configure correct value of max uni-streams in QUIC profile.
Component: TMOS
Symptoms:
After booting the system, you notice zombie 'rm' processes:
$ top -b | awk '$8=="Z"'
14461 root 20 0 0 0 0 Z 0.0 0.0 0:00.00 rm
14461 root 20 0 0 0 0 Z 0.0 0.0 0:00.00 rm
14461 root 20 0 0 0 0 Z 0.0 0.0 0:00.00 rm
Restarting tmipsecd will kill the zombied process but will start a new one.
Conditions:
-- IPsec is enabled.
-- Booting up the system.
Impact:
A zombie 'rm' process exists. There should be no other impact.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
The HTTP/2 Gateway configuration is used without the HTTP MRF Router.
The TMM may crash in rare scenarios when a stream is being torn down.
Conditions:
-- HTTP/2 is configured in the Gateway scenario.
-- The HTTP MRF Router is not used.
Impact:
Traffic disrupted while tmm restarts.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
When multiple database servers are monitored by LTM database (MSSQL, MySQL, PostgreSQL, Oracle) monitors and one database server goes down (such by stopping the database server process), a deadlock may occur in the LTM database monitor daemon (DBDaemon) which causes an interruption in monitoring of other database servers.
When this occurs, one database server going down may cause all monitored database servers to be marked Down for several minutes until the blocking operation times out and normal monitoring can resume.
Conditions:
This may occur when:
1. Running a version of BIG-IP or an Engineering Hotfix which contains fixes for bugs ID769309 and ID775901.
2. Stopping a monitored database server process (such as by halting the database service).
Impact:
Monitoring of database servers may be interrupted for up to several minutes, causing monitored database servers to be marked Down. This may persist for several minutes until the blocking operation times out, the backlog of blocked DB monitor threads are processed to completion, and normal DB monitoring resumes.
Workaround:
You can prevent this issue from occurring by using a different LTM monitor type (such as a TCP monitor or external monitor) to monitor the database servers.
Component: Local Traffic Manager
Symptoms:
QUIC gets lower goodput compared to TCP when tested on lossy links.
Conditions:
The tested links are lossy (e.g, 0.1% loss probability).
Impact:
QUIC completes the data transfer in longer time.
Workaround:
QUIC achieves similar or better goodput compared to TCP on lossy links.
Component: Local Traffic Manager
Symptoms:
QUIC rate pacing becomes dis-functional under some conditions.
Conditions:
- QUIC rate pacing is in use.
- Packet size becomes slightly larger than available rate pacing bytes.
Impact:
QUIC rate pacing becomes noneffective which leads to sending data more bursty.
Workaround:
QUIC rate pacing does not become dis-functional under some conditions anymore.
Component: Local Traffic Manager
Symptoms:
When using the global SNAT feature on LTM, IPv4 TTL/IPv6 Hop-Limit values may be erroneously set to 255/64 on egress.
Conditions:
Traffic is handled by global SNAT.
Impact:
TTL on egress is set to 255/; Hop-Limit on egress is set to 64.
Workaround:
None.
Component: TMOS
Symptoms:
After rebooting BIG-IP Virtual Edition (VE) deployed on a cloud provider, the instance enters LICENSE INOPERATIVE state.
Errors similar to one below are seen in an LTM log:
err chmand[4770]: Curl request to metadata service failed with error(7): 'Couldn't connect to server'.
Conditions:
- Static management IP address configuration.
- Instance is restarted.
Impact:
Instance is not operational after restart.
Workaround:
After instance is fully booted, reload the license with 'reloadlic'.
Component: Application Security Manager
Symptoms:
Non-ASCII parameter static values are garbled when created in a non-UTF-8 policy using REST.
Conditions:
-- The policy is configured for an encoding other than UTF-8.
-- Attempting to create non-ASCII parameter static values using REST.
Impact:
Parameter static values containing non-ASCII characters are garbled when created using REST.
Workaround:
Use UTF-8.
Component: Application Security Manager
Symptoms:
-- The bd_agents stops sending the configuration in the middle of startup or a configuration change.
-- The policy may be incomplete in the bd causing incorrect enforcement actions.
Conditions:
This is a rarely occurring issue, and the exact conditions that trigger it are unknown.
Impact:
-- The bd_agent hangs or restarts, which may cause a complete ASM restart (and failover).
-- A partial policy may exist in bd causing improper enforcement.
Workaround:
-- Unassign and reassign the policy.
-- if unassign/reassign does not help, export and then reimport the policy.
Component: Global Traffic Manager (DNS)
Symptoms:
During config load or system start-up, you see the following error:
01071769:3: Decryption of the field (privatekey) for object (13079) failed.
Unexpected Error: Loading configuration process failed.
Conditions:
-- TSIG keys are present in the device configuration.
-- The device's master key is changed.
Impact:
Unable to view TSIG keys. Configuration cannot be loaded.
Workaround:
None.
Component: Access Policy Manager
Symptoms:
Tmm memory usage keeps going up when passing multiple HTTP requests through a kept-alive TCP connection carrying an OAuth token as bearer in the Authorization header.
Conditions:
-- Multiple HTTP requests through a kept-alive TCP connection.
-- Requests carry an OAuth token as bearer in the Authorization header.
Impact:
Memory leak occurs in which tmm memory usage increases.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
First HTTP request on a connection creates a connection to the server. Subsequent pipelined requests should be processed and use the established connection to the server. However, the requests were queued in the proxy and not processed resulting in connection hang.
Conditions:
- HTTP or HTTP/2 virtual server with httprouter profile.
- HTTP/1.1 connections with the client and server.
- Pipelined HTTP requests.
Impact:
Connection hangs and is eventually reset.
Workaround:
No workaround.
Component: TMOS
Symptoms:
After upgrading to v14.1.2.1, logs are not generated and sysstat.service is not running.
Conditions:
-- Upgrade from BIG-IP v12.1.x (which uses CentOS 6) to BIG-IP v14.1.2.1 or later (which uses CentOS 7).
-- The issue is momentary and is not always reproducible.
Impact:
Logs are not generated and sysstat.service is not running.
Workaround:
Once the BIG-IP system starts up, check for failed services:
systemctl list-units --failed
If results show sysstat.service as FAILED, run the following command:
restorecon -Rv /var/log/sa6 && systemctl start sysstat
Component: TMOS
Symptoms:
/var/log/message and /var/log/user.log files have lines that only contain asterisks.
For example:
Nov 12 10:40:57 bigip1 **********************************************
Conditions:
Snmp query an OID handled by sflow, for example:
snmpwalk -v2c -c public localhost SNMPv2-SMI::enterprises.14706.1.1.1
Impact:
The impact is cosmetic only, however it could make reading the logs more difficult if the sflow snmp tables are constantly being queried.
Workaround:
You have two options:
-- Filter out all sflow_agent log messages
-- Filter out all messages that contain a newline '\n' or carriage return character '\r'.
Both workarounds are done by editing the syslog template, this means that if the you upgrades, you must edit the template again to reinstate the workaround.
=============================================
Solution #1 - Filter out all sflow_agent logs:
1) remount /usr as read+write:
mount -o rw,remount /usr
2) Make a backup copy of the template:
cp /usr/share/defaults/config/templates/syslog.tmpl /usr/share/defaults/config/templates/syslog.tmpl.orig
3) Add write permissions to the template:
chmod +w /usr/share/defaults/config/templates/syslog.tmpl
4) Add the filter to syslog.tmpl
4a) Open syslog.tmpl for edit:
vi /usr/share/defaults/config/templates/syslog.tmpl
4b) Add the new filter after the filter f_messages:
filter f_not_sflow {
not match ("sflow_agent" value("$PROGRAM"));
For example:
filter f_messages {
level(UNIX_CONFIG_SYSLOG_REPLACE_MESSAGESFROM..UNIX_CONFIG_SYSLOG_REPLACE_MESSAGESTO)
and not facility(auth, authpriv, cron, daemon, mail, news)
and not message("WA");
filter f_not_sflow {
not match ("sflow_agent" value("$PROGRAM"));
4c) Add the filter to the log that sends all source local message to the syslog pipe:
log {
source(local);
filter(f_not_sflow);
destination(d_syslog_pipe);
5) Save the changes and quit vi.
6) In order for the BIG-IP system to write out the syslog conf with the modified template, you must change the syslog configuration. To do so, use tmsh to modify the 'daemon-from' to 'info' and then back to the default of 'notice':
tmsh modify /sys syslog daemon-from info
tmsh modify /sys syslog daemon-from notice
7) Ensure the changes were written to /etc/syslog-ng/syslog-ng.conf.
8) remount /usr as read-only
mount -o ro,remount /usr
=============================================
Solution #2 - Filter out all messages with \n or \r:
1) remount /usr as r+w:
mount -o rw,remount /usr
2) Make a backup copy of the template:
cp /usr/share/defaults/config/templates/syslog.tmpl /usr/share/defaults/config/templates/syslog.tmpl.orig
3) Add write permissions to the template:
chmod +w /usr/share/defaults/config/templates/syslog.tmpl
4) Add the filter to syslog.tmpl:
4a) Open syslog.tmpl for edit:
vi /usr/share/defaults/config/templates/syslog.tmpl
4b) Add the new filter after the filter f_messages:
filter f_no_multi_line {
not (message('\n') or message('\r'));
For example:
filter f_messages {
level(UNIX_CONFIG_SYSLOG_REPLACE_MESSAGESFROM..UNIX_CONFIG_SYSLOG_REPLACE_MESSAGESTO)
and not facility(auth, authpriv, cron, daemon, mail, news)
and not message("WA");
filter f_no_multi_line {
not (message('\n') or message('\r'));
4c) Add the filter to the log that sends all source local message to the syslog pipe:
log {
source(local);
filter(f_no_multi_line);
destination(d_syslog_pipe);
5) Save the changes and quit vi.
6) In order for the BIG-IP system to write out the syslog conf with the modified template, you must change the syslog configuration. To do so, use tmsh to modify the 'daemon-from' to 'info' and then back to the default of 'notice':
tmsh modify /sys syslog daemon-from info
tmsh modify /sys syslog daemon-from notice
7) Ensure the changes were written to /etc/syslog-ng/syslog-ng.conf.
8) remount /usr as read-only:
mount -o ro,remount /usr
Component: Application Security Manager
Symptoms:
Global/URL/Flow Parameters with flag is_sensitive true are not masked in referrer and their value may be exposed in logs.
Conditions:
Global/URL/Flow Parameters with flag is_sensitive true are defined in the policy. In logs, the value of such parameter will be masked in QS, but will be exposed in the referrer.
Impact:
The parameter will not be masked in 'Referer' value header in logs, although it is masked in 'QS' string.
Workaround:
Can defined the parameters as global sensitive parameters.
Component: Local Traffic Manager
Symptoms:
-- High tmm CPU utilization.
-- Stalled connections.
-- Incorrect persistence decisions.
Conditions:
-- A OneConnect profile is combined with certain persist profiles on a virtual server.
-- The virtual server configuration is changed while there is an ongoing connection to the virtual server. Any connections that make a request after the configuration change can be affected.
-- The persistence types that are affected are:
- Source Address (but not hash-algorithm carp)
- Destination Address (but not hash-algorithm carp)
- Universal
- Cookie (only cookie hash)
- Host
- SSL session
- SIP
- Hash (but not hash-algorithm carp)
Impact:
-- High tmm CPU utilization.
-- Stalled connections.
-- Incorrect persistence decisions.
Workaround:
None.
Component: Global Traffic Manager (DNS)
Symptoms:
Running gtm_add failed to restart the named daemon.
Conditions:
Run gtm_add to completion.
Impact:
Named is not restarted. No BIND functionality.
Workaround:
Restart named:
bigstart start named
Component: TMOS
Symptoms:
When a secondary blade is a new blade or is booted without a binary db, the LLDP settings on the blade's interface is reset to default.
Conditions:
Plug in a new secondary blade, or reboot a blade (that comes up as secondary) without a binary db.
Impact:
The flow-control setting is reset to default (tx-rx).
Workaround:
Reload the configuration on the primary blade.
Component: TMOS
Symptoms:
The icrd returns an incorrect route names when a '.' (dot, or period) is present in the subPath, as it treats the subPath as an IP address and the leaf name as a subnet and considers its name as a whole. Also the subPath field is missed in the response route object. This happens only in the case of curl request.
Conditions:
-- The subPath contains a '.' in it.
-- A curl request is made.
Impact:
Result information is not compatible with actual result.
Workaround:
None.
Component: Application Security Manager
Symptoms:
Live Update of Browser Challenges fails installation.
Live Update provides an interface on the F5 Downloads site to manually install or configure automatic installation of various updates to BIG-IP ASM components, including ASM Attack Signatures, Server Technologies, Browser Challenges, and others.
Conditions:
-- From the F5 Downloads side, select a software version.
-- Click BrowserChallengesUpdates.
-- Attempt to download and install Download BrowserChallenges<version_number>.im.
Note: Browser Challenges perform browser verification, device and bot identification, and proactive bot defense.
Impact:
Browser Challenges update file cannot be installed.
Workaround:
None.
Component: Access Policy Manager
Symptoms:
SELinux policy does not allow the rewrite plugin to create a directory and write troubleshooting data into /var/tmp/WebAppTrace.
Conditions:
Collecting Portal Access web applications traces per K13384: Performing a web applications trace (11.x - 14.x) :: https://support.f5.com/csp/article/K13384
Impact:
Cannot collect Portal Access web applications troubleshooting data as it described in in that AskF5 Article.
Workaround:
Connect via SSH using the root account and run this command:
restorecon -Rv /var/tmp/WebAppTrace/
Component: Global Traffic Manager (DNS)
Symptoms:
You might see occasional and temporary CPU usage increases when named.conf file is unusually large.
Conditions:
Unusually large named.conf file and zones are checked for updates (when the SOA expires).
Impact:
When a zone file is updated, a downstream effect is the ZoneRunner process to parse again the named.conf file. The parsing of an unusually large file may cause a temporary increase in CPU usage.
Workaround:
None.
Component: Access Policy Manager
Symptoms:
Per-request policy cannot be exported if it contains an HTTP Connector agent.
Conditions:
-- Create a Per Request Policy.
-- In the sub-routine section, create a new sub-routine and
attach HTTP Connector to that sub-routine.
-- After the policy creation is done, export the policy.
Impact:
Per-request policy cannot be exported and reports an error.
Workaround:
None.
Component: Access Policy Manager
Symptoms:
The log message when OAuth client or resource server fails to generate the secret is assigned an incorrect log level, and is incorrectly logged at the emergency level.
Conditions:
This is encountered when this message is logged by mcpd.
Impact:
Log message cannot be grouped with messages at the correct log level.
Workaround:
None.
Component: Access Policy Manager
Symptoms:
Java applets are not patched when accessed through APM Portal Access.
/var/log/rewrite contains error messages similar to following:
-- notice rewrite - fm_patchers/java_patcher_engine/CryptoToolsManager.cpp:568 (0x1919ab0): CryptoToolsManager :: _ReadCA() - cannot open CA file.
/var/log/auditd/audit.log contains AVC denials for rewrite on attempt to read file under /config/filestore/.
Conditions:
Java patching is enabled via rewrite profile and Portal Access resource.
Impact:
Java applets cannot be patched by APM Portal Access rewriter.
Workaround:
None.
Component: TMOS
Symptoms:
With multiple virtual servers that have the same destination address, changing all of them in the GUI to use an address list as their source address will result in the last one changed failing with an error similar to:
01070344:3: Cannot delete referenced virtual address /Common/1.2.3.4.
Conditions:
-- More than one virtual server with the same destination address.
-- Changing all the virtual servers that share the same destination address to use an address list for their source address.
Impact:
Unable to change the source address of a virtual server to an address list.
Workaround:
Use TMSH to manually create a traffic-matching criteria object and assign it to the virtual server:
tmsh create ltm traffic-matching-criteria <virtual server name>_VS_TMC_OBJ destination-address-inline <destination address of virtual server> destination-port-inline <destination port of virtual server> source-address-inline 0.0.0.0 source-address-list <address list name>
tmsh modify /ltm virtual <virtual server name> traffic-matching-criteria <virtual server name>_VS_TMC_OBJ destination 0.0.0.0:any
Component: Application Security Manager
Symptoms:
Signature Update installations using ASU files with the same name on versions before 14.1.0 block Live Update usage after upgrade to 14.1.0 or later.
Conditions:
The same Signature Update file is installed multiple times on the device when running a version earlier than 14.1.0.
Impact:
Signature Update cannot be installed using Live Update, and errors appear in logs.
Workaround:
1. Delete the file: /var/lib/hsqldb/live-update/live-update-import.yaml.
2. Restart tomcat:
bigstart restart tomcat
This causes pre-upgrade records for Signature Update to be lost, but does not have any other functional impact.
** Another Workaround incase the above does not solve the issue:
1. stop the tomcat server:
> bigstart stop tomcat
2. clean the live update db :
> cd /var/lib/hsqldb/live-update
> rm -f liveupdatedb.*
3. remove all *.im files which are not genesis file (factory-default files):
3.1 get the list of genesis files:
> less /etc/live-update-genesis.yaml | grep genes | cut -d":" -f2
3.2 go to update file directory:
> cd /var/lib/hsqldb/live-update/update-files
3.3 manually remove the *.im files that are not genesis
4. restart the tomcat server:
> bigstart start tomcat
Component: TMOS
Symptoms:
The vmxnet3 driver cannot handle MTUs larger than 9000 bytes.
Conditions:
-- Using a BIG-IP Virtual Edition (VE) with the vmxnet3 driver.
-- Passing packets larger than 9000 bytes.
Impact:
Either packets are dropped, or the hypervisor may crash on some platforms that do not handle this condition gracefully.
Workaround:
Modify the tmm_init.tcl file, adding the following line:
ndal mtu 9000 15ad:07b0
Component: Policy Enforcement Manager
Symptoms:
When sessions usage iRules are called on an already overloaded system it might crash.
Conditions:
Session iRule calls on heavily overloaded BIG-IP systems.
Impact:
Tmm restarts. Traffic disrupted while tmm restarts.
Workaround:
Reduce the load on tmm or modify the optimize the irule.
Component: Local Traffic Manager
Symptoms:
Under undetermined load pattern TMM may crash with message: Assertion 'valid node' fail.
Conditions:
This can occur while passing traffic with the Ram Cache profile enabled on a Virtual Server. Other conditions are unknown.
Impact:
Traffic disrupted while tmm restarts.
Workaround:
Refrain from using ramcache may mitigate the problem.
Component: TMOS
Symptoms:
Mac addresses are forced to be the same for ixlv trunks.
Conditions:
This happens when ixlv trunks are used.
Impact:
Mac addresses may not be as depicted on the device.
Workaround:
None.
Component: Service Provider
Symptoms:
When SIP message routing fails to route to a pool member (Triggering a MR_FAILED, MR::message status of 'no connection'), The BIG-IP system caches the failed state and continues to return this even after the pool member becomes reachable again.
Conditions:
When BIG-IP systen fails to route messages to the peer (server) due to unavailability of route or any other issues.
Impact:
The BIG-IP system is never be able to establish connection to the peer.
Workaround:
None.
Component: Application Security Manager
Symptoms:
Browser Challenges default installation fails in 15.1.0 after upgrade or resetting back to default.
BIG-IP software v15.1.0 ships with a BrowserChallenges_20191121_043810.im file that does not have a proper encryption, and when trying to install the file via the Live Update page the following error occurs:
gpg: WARNING: unsafe ownership on homedir `/ts/share/negsig/gpg_asm_sigfile_installer'
gpg: encrypted with 1024-bit ELG key, ID 7C3E3CE5, created 2007-03-20
"asm_sigfile_installer"
gpg: Signature made Thu 21 Nov 2019 02:38:10 PM IST using RSA key ID BC67BA01
gpg: Can't check signature: No public key
Conditions:
Live Update file BrowserChallenges_20191121_043810.im has a different status than 'Currently Installed'.
Impact:
If the file 'BrowserChallenges_20191121_043810.im ' is the newest file then upgrade is not applicable.
Workaround:
Browser Challenges update file can now be installed via Live Update.
Component: TMOS
Symptoms:
When an outgoing SCTP connection is created using an ephemeral port, the connection may appear to be open after an SCTP connection halt. This prevents new connections to the same endpoint, as the connection appears to already exist.
Conditions:
-- A virtual server configured with an SCTP profile.
-- An outgoing SCTP connection after an existing connection to the same endpoint has halted.
Impact:
New connections are unable to be created resulting in dropped messages.
Workaround:
None.
Component: TMOS
Symptoms:
When the system transitions from active or next active (standby), e.g., to offline, the internal flow of a tunnel can be expired.
If the device returns to active or standby, and if the tunnel is modified, a double flow removal can cause a tmm crash.
Conditions:
-- System transitions from active or next active.
-- Tunnel is modified.
-- Device returns to active or next active mode.
Impact:
The tmm process restarts. Traffic disrupted while tmm restarts.
Workaround:
None.
Component: TMOS
Symptoms:
Tmm receives an update from the ePVA for a hardware accelerated connection that is matched to the wrong correction. This can result in a tmm core, which is reported as a segment fault in the tmm log files.
Conditions:
A FastL4 virtual server that has hardware acceleration enabled.
Impact:
Traffic disrupted while tmm restarts.
Workaround:
Disable hardware acceleration.
Component: TMOS
Symptoms:
The license installation and activation process takes a very long time to complete.
Conditions:
- BIG-IP Virtual Edition running on Google Compute Engine (GCE) Platform.
- Activating the BIG-IP license.
Impact:
It can take 2-3 minutes to report the device is licensed and 3-4 minutes for BIG-IP system to become Active after that.
Workaround:
None.
Component: TMOS
Symptoms:
TMM may crash when a tunnel is used after the unit returns from offline status.
Conditions:
-- Tunnel is configured and active.
-- Unit is transitioned from offline to online.
-- Tunnel is used after online status is restored.
Impact:
TMM crashes. Traffic disrupted while tmm restarts.
Workaround:
None.
Component: Application Visibility and Reporting
Symptoms:
The HTTP/2 version appears in charts, but when clicking on the chart reports, errors are reported in monpd log and the chart is empty in the GUI, with an error reported in the GUI and in the monpd log:
-- DB|ERROR|Oct 21 06:12:24.578|22855|../src/db/MonpdDbAPI.cpp:mysql_query_safe:0209| Error (err-code 1054) executing SQL string :
-- DB|ERROR|Oct 21 06:12:24.578|22855|../src/db/MonpdDbAPI.cpp:runSqlQuery:0677| Error executing SQL query:
-- REPORTER|ERROR|Oct 21 06:12:24.578|22855|../src/reporter/handlers/ReportRunnerHandler.cpp:runReport:0409| Results for query came back as NULL
-- REPORTER|ERROR|Oct 21 06:12:24.578|22855|../src/reporter/ReporterUtils.cpp:throwInternalException:0105| throwing exception to client error code is 1 error msg is Internal error
Conditions:
-- Create a new policy or use an existing policy.
-- Go to Security :: Reporting : Application : Charts.
-- Select weekly charts.
Impact:
Charts are empty in the GUI, and the system logs errors in monpd.
Workaround:
None.
Component: Service Provider
Symptoms:
SCTP filter fails to create outgoing connections if the peer requests multihoming. The failure may produce a tmm core.
Conditions:
Usage of SCTP multi-homing with a MRF transport-config.
Impact:
The outgoing connection is aborted or tmm may core. Traffic disrupted while tmm restarts.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
When a user tries to create a virtual server with address list, it might fail with below shown error:
01b90011:3: Virtual Server /Common/VS1's Traffic Matching Criteria /Common/testvs1 illegally shares destination address, source address, service port, and ip-protocol with Virtual Server /Common/testvs2 destination address, source address, service port.
Conditions:
-- One or more virtual servers that were created via the GUI already exist on the BIG-IP system.
-- Attempt to use the GUI to create another virtual server with address list.
Impact:
Cannot create the virtual server.
Workaround:
Create the virtual server via tmsh:
-- First create the traffic matching criteria using the address list.
-- Then use the traffic matching criteria to create a virtual server.
Component: Device Management
Symptoms:
Rolling restarts of restjavad every few seconds typically due to failure to start and reports messages in daemon log:
daemon.log: emerg logger: Re-starting restjavad
The system reports similar message at the command line.
No obvious cause is logged in rest logs.
Conditions:
-- System DB variable provision.extramb has an unusually high value*:
+ above ~2700-2800MB for v12.1.0 and earlier.
+ above ~2900-3000MB for v13.0.0 and later.
-- On v13.0.0 and later, sys db variable restjavad.useextramb needs to have the value 'true'
*A range of values is shown. When the value is above the approximate range specified, constant restarts are extremely likely, and within tens of MB below that point may be less frequent.
To check the values of these system DB varaiables use:
tmsh list sys db provision.extramb
tmsh list sys db restjavad.useextramb
Impact:
This impacts the ability to use the REST API to manage the system
Workaround:
If needing sys db restjavad.useextramb to have the value 'true', keep sys db provision.extramb well below the values listed (e.g., 2000 MB work without issue).
To set that at command line:
tmsh modify sys db provision.extramb value 2000
If continual restarts of restjavad are causing difficulties managing the unit on the command line:
1. Stop restjavad (you can copy this string and paste it into the command line on the BIG-IP system):
tmsh stop sys service restjavad
2. Reduce the large value of provision.extramb if necessary.
3. Restart the restjavad service:
tmsh start sys service restjavad
Component: Local Traffic Manager
Symptoms:
Gratuitous ARPs (GARPs) are not sent out when moving a virtual-address from one floating traffic-group to another (e.g., from traffic-group-1 to traffic-group-2).
Conditions:
-- Moving a virtual-address from one floating traffic-group to another.
-- The traffic-groups are active on different devices.
Impact:
Application traffic does not immediately resume after the virtual-address is moved. Instead, the surrounding network devices have to ARP out for the IP address after reaching a timeout condition.
Workaround:
After moving the virtual-address, disable and then re-enable the ARP setting for the virtual-address. This forces GARPs to be sent out.
Component: Local Traffic Manager
Symptoms:
BIG-IP may set TTL to 255 on forwarded packets.
Conditions:
Virtual-server with ipother profile and SNAT configured.
Impact:
Traffic leaves with egress TTL set to 255.
Workaround:
None.
Component: TMOS
Symptoms:
The HSB hardware returns an invalid receive (rx) descriptor to TMM. This results in a TMM core and can be seen as a SIGSEGV in the TMM logs. This also might result in continuous restarts of TMM, resulting in subsequent SIGSEGVs reported in the TMM logs until the unit is manually recovered.
Conditions:
The exact conditions under which this occurs are unknown.
Impact:
Traffic disrupted while tmm restarts. This may result in continuous TMM restarts until the BIG-IP system is rebooted.
Workaround:
None.
Component: Access Policy Manager
Symptoms:
TMM might crash due to SIGABRT.
Conditions:
-- Session check agent is present in APM per-request policy.
-- APM Access Profile scope changes during SSL Orchestrator upgrade.
-- This issue can occur for SSL Orchestrator upgrades from 14.x to 15.x and above.
Impact:
TMM might crash once. Traffic disrupted while tmm restarts.
Workaround:
None.
Component: Application Visibility and Reporting
Symptoms:
If collect-all-dos-statistic is enabled, AVR 'promises' to the client a JavaScript injection in the response by adding the expected length of the JavaScript to the Content-length header.
Conditions:
Security :: reporting : settings : collect-all-dos-statistic is enabled.
Impact:
In addition to collecting DoS statistics, JavaScript injection also occurs.
Workaround:
Can use iRules to control which pages should get the JavaScript injection.
For detailed information, see K13859: Disabling CSPM injection with iRules :: https://support.f5.com/csp/article/K13859.
Component: Application Visibility and Reporting
Symptoms:
'Undefined entity dosl7_vip was used' error message is reported on widgets whenever a 'Virtual Server' filter is selected on the 'Security :: Reporting : DoS : Custom Page' GUI page.
Conditions:
-- Navigate to Security :: Reporting : DoS : Custom Page in the GUI.
-- Filter widgets results with specific 'Virtual Server'.
Impact:
GUI widgets report errors and cannot show stats.
Workaround:
This GUI fix requires modifying a single PHP file in one location, which you can do directly on your BIG-IP system with a few bash commands:
1. Backup the file '/var/ts/dms/amm/common/ovw/dos_custom_overview_commons.php':
$ cp /var/ts/dms/amm/common/ovw/dos_custom_overview_commons.php /shared/
2. Change permissions to allow modifying it:
$ chmod +w /var/ts/dms/amm/common/ovw/dos_custom_overview_commons.php
3. Change the file to include the fix:
$ sed -i 's/dosl7_vip/vip/g' /var/ts/dms/amm/common/ovw/dos_custom_overview_commons.php
$ sed -i "s/ANALYTICS_MOD_DNS_DOS => 'vip'/ANALYTICS_MOD_DNS_DOS => 'dns_vip'/g" /var/ts/dms/amm/common/ovw/dos_custom_overview_commons.php
4. Verify that the fix is as expected:
$ vimdiff /var/ts/dms/amm/common/ovw/dos_custom_overview_commons.php /shared/dos_custom_overview_commons.php
(** You should see two lines modified:
1. ANALYTICS_MOD_DOSL7 => 'dosl7_vip' to ANALYTICS_MOD_DOSL7 => 'vip'.
2. ANALYTICS_MOD_DNS_DOS => 'vip' to ANALYTICS_MOD_DNS_DOS => 'dns_vip')
5. Revert permissions of the file:
$ chmod -w /var/ts/dms/amm/common/ovw/dos_custom_overview_commons.php
6. Log out and log back into the GUI, so that the new version of the file loads.
Component: TMOS
Symptoms:
Under certain conditions, the BIG-IP system requires you to change your password on every login.
Furthermore, the login then fails, and loops endlessly asking for the password, even though the password has not expired.
Conditions:
-- BIG-IP 14.0.0 and later.
-- LDAP authentication is used for remote users.
-- Active Directory (AD) user account has shadowLastChange attribute with a value of 0 (or anything lower than the number of days since 1-1-1970).
Impact:
Remote AD BIG-IP users are unable to login to the BIG-IP system using remote LDAP authentication
Workaround:
Clear the value of shadowLastChange within AD.
Component: Global Traffic Manager (DNS)
Symptoms:
Configuration fails to load after upgrade with a message:
01420006:3: Can't find specified cli schema data for x.x.x.x
Where x.x.x.x indicates an older version of BIG-IP software than is currently running.
Conditions:
-- Orphaned bigip_gtm.conf from an older-version. This can occur if GTM/DNS is provisioned, then deprovisioned before upgrade, leaving behind a bigip_gtm.conf with the old schema.
-- Upgrading to a new version that does not contain the schema for the old version that the bigip_gtm.conf uses.
Impact:
Configuration fails to load after upgrade.
Workaround:
Before upgrading:
If the configuration in bigip_gtm.conf is not needed, then it can be renamed (or deleted) before upgrading:
mv /config/bigip_gtm.conf /config/bigip_gtm.conf.id837637
tmsh load sys config gtm-only
After upgrading (i.e., with the system in the Offline state) services must be restarted to pick up the change:
mv /config/bigip_gtm.conf /config/bigip_gtm.conf.id837637
tmsh restart sys service all
Component: Service Provider
Symptoms:
In MBLB/SIP, if the BIG-IP system attempts to send messages to the destination over a TCP connection that is in FIN-wait2 stage, instead of returning a failure and silently dropping the message, the BIG-IP system attempts to create a new TCP connection by sending a SYN. Eventually, the attempt fails and causes the connection to be aborted.
Conditions:
-- This happens on MBLB/SIP deployment with TCP.
-- There is message sent from the server to the BIG-IP system.
-- The BIG-IP system forwards the message from the server-side to client-side.
-- The destination flow (for the BIG-IP system to forward the message to) is controlled by 'node <ip> <port>' and 'snat <ip> <port>' iRules command.
-- The destination flow is in the FIN-wait2 stage.
Impact:
This causes the BIG-IP system to abort the flow that originates the message.
Workaround:
None.
Component: Application Visibility and Reporting
Symptoms:
Adding SMTP config to default HTTP analytics profile results in config parsing failures for child profiles that are assigned to virtual servers. Removing SMTP config resolves the issue. The 'tmsh load sys config' command fails with the following error:
-- 01020036:3: The requested profile (/Common/child-analytics) was not found.
-- Unexpected Error: Validating configuration process failed.
Conditions:
-- Child analytics profile applied to virtual server.
-- Parent analytics profile contains SMTP config.
Impact:
Loading configuration might fail.
Workaround:
None.
Component: Access Policy Manager
Symptoms:
Some text in APM Server pages, such as the logout page, are presented in English even when using a different language.
Conditions:
Use APM with a localized language, and certain strings for pages like logout, Webtop, or EPS, would still be in English.
Impact:
Some strings are displayed in English instead of localized language.
Workaround:
None.
Component: TMOS
Symptoms:
Some onboarding features are not available in the current version of walinuxagent.
Conditions:
Attempting to use a feature that is not available in the current version of the Azure walinuxagent that is included in the BIG-IP release.
Impact:
Cannot use new features in the Azure walinuxagent until the Azure walinuxagent is updated.
Workaround:
None.
Component: Application Security Manager
Symptoms:
Idle async handlers can remain loaded for a long time doing nothing because they do not have an idle timer. The sum of such idle async handlers can add unnecessary memory pressure.
Conditions:
This issue might result from several sets of conditions. Here is one:
Exporting a large XML ASM policy and then leaving the BIG-IP system idle. The relevant asm_config_server handler process increases its memory consumption and remains that way, holding on to the memory until it is released with a restart.
Impact:
Depletion of memory by lingering idle async handlers may deprive other processes of sufficient memory, triggering out-of-memory conditions and process failures.
Workaround:
-- Restart asm_config_server, to free up all the memory that is currently taken by all asm_config_server processes and to impose the new MaxMemorySize threshold:
---------------
# pkill -f asm_config_server
---------------
-- Restart asm_config_server periodically using cron, as idle handlers are soon created again.
Component: Access Policy Manager
Symptoms:
Category Lookup agent has changed to include the IP in the categorization query. The BIG-IP TMUI does not do the same (Access Policy :: Secure Web Gateway : Database Settings : URL Category Lookup).
Conditions:
-- SWG or URLDB provisioned.
-- Run traffic with category lookup in the PRP and note the category produced.
-- Run the same URL through the GUI lookup tool or the command line tool.
Impact:
Some websites may be categorized differently depending on if the IP address is passed in or not.
Workaround:
None.
Component: Access Policy Manager
Symptoms:
F5 Endpoint Inspection helper app is not updated, but other components such as F5 VPN helper App is auto updated.
Conditions:
Use a browser to establish VPN
Impact:
End users cannot to receive bug fixe or feature enhancement updates.
Workaround:
Download and install F5 Endpoint Inspection helper from BIG-IP.
https://APM_SERVER/public/download/f5epi_setup.exe
Component: Access Policy Manager
Symptoms:
When the URL being accessed exceeds a length of 8 KB, the BIG-IP resets the connection.
Conditions:
-- APM deployed with a per-request policy.
-- The per-request policy includes a category lookup.
Impact:
The APM end-user connection is reset, and the system posts an error message in /var/log/apm:
-- crit tmm[23363]: 01790601:2: [C] 10.62.118.27:65343 -> 65.5.55.254:443: Maximum URL size exceeded.
Workaround:
None.
Component: Access Policy Manager
Symptoms:
Both AD Query and LDAP Auth/Query fails.
Conditions:
-- AD Query Agent, LDAP Auth Agent, or LDAP Query Agent is configured in Per-Session or Per-Request Policy.
-- These agents are configured in Direct mode.
-- The AD and LDAP server address is configured as IPv6 address.
Impact:
Users may not be able to login to APM, and hence service is disrupted.
Workaround:
None.
Component: TMOS
Symptoms:
SNMP get requests for ipAddr, ipAddress, ipAddressPrefix and ipNetToPhysical are slow to respond.
Conditions:
Using SNMP get requests for ipAddr, ipAddress, ipAddressPrefix and ipNetToPhysical.
Impact:
Slow performance.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
Virtual wire is forwarding traffic on standby resulting in traffic loops and potential network outage.
Conditions:
-- High availability (HA) configured.
-- Virtual wire configured.
Impact:
Standby device is passing traffic, which may create traffic loops and bring down the network.
Workaround:
Do not configure virtual wire on standby devices.
Component: TMOS
Symptoms:
When resizing the appdata logical volume, the change may not be honored. This is because sometimes the disk metadata does not support the change without unmounting and remounting the disk.
Conditions:
This issue applies to deployments that provision multiple modules requiring a large appdata logical volume.
Impact:
The appdata logical volume cannot be resized, so you must reduce the number of modules and the associated provisioning level so that the existing appdata logical volume size does support them.
Workaround:
None.
Component: Application Visibility and Reporting
Symptoms:
Avrd crashes, one or more core avrd files exist in /var/core
Conditions:
-- A BIG-IP system is managed by BIG-IQ via secure channel
-- Avrd is restarted.
Impact:
Avrd cores as it is shutting down. During avrd shutdown, the BIG-IQ data collection device (DCD) is unreachable for 10 minutes
Workaround:
None.
Component: TMOS
Symptoms:
The /var partition might become completely full on the disk due to tmp files being written to /var/config/rest. This condition may be accompanied by console error messages similar to the following:
011d0004:3: Disk partition /var (slot #) has only 0% free on secondary blade.
Additionally, there may be periodic restjavad and bigd daemon restarts related to disk space exhaustion.
Conditions:
Process traffic while DoS Dashboard is open.
This issue is happening because a VIPRION process is not available because of a REST timeout.
Impact:
The partition housing /var/config/rest may become 100% full, impacting future disk IO to the partition.
Workaround:
Manually run the following commands, in sequence:
bigstart stop restjavad
rm -rf /var/config/rest/*.tmp
bigstart start restjavad
Component: TMOS
Symptoms:
When multiple users are issuing REST commands, memory may leak slowly in icrd_child.
Conditions:
-- The icrd_child process is running.
-- There are multiple users accessing device via REST.
Impact:
Memory slowly leaks in icrd_child.
Workaround:
None.
Component: TMOS
Symptoms:
-- The iControl REST commands respond with the following:
[INFO] Text: u'{"code":200,"message":"REST system unavailable due to disk corruption! See /var/log/restjavad.*.log for errors.","restOperationId":1472895,"kind":":resterrorresponse"}'
-- The GUI indicates that iAppLX sub-system is unresponsive.
-- On the BIG-IP device, /var/config/rest/storage/LOST-STORAGE.txt exists.
Conditions:
The conditions that trigger this are unknown. It might be due to a previous catastrophic event such as power loss or out-of-memory errors.
Manually creating the file /var/config/rest/storage/LOST-STORAGE.txt can also trigger this error.
Impact:
The iControl REST system is unavailable.
Workaround:
Run the following commands at the BIG-IP command prompt:
bigstart stop restjavad restnoded
rm -rf /var/config/rest/storage
rm -rf /var/config/rest/index
bigstart start restjavad restnoded
rm -f /var/config/rest/downloads/*.rpm
rm -f /var/config/rest/iapps/RPMS/*.rpm
tmsh restart sys service tomcat
Then, reinstall any iAppLX packages that were installed.
Component: Application Visibility and Reporting
Symptoms:
Systems with a large amount of statistics data collected in the local database (i.e., systems not working with BIG-IQ) can have high IO Wait CPU usage, peaking at 10 minutes, 1 hour, and 24 hours. This is caused by the data aggregation process that is running on the local database. Notice that large memory footprints, particularly for avrd might be a symptom for the phenomenon.
Conditions:
-- The BIG-IP system is collecting statistics locally (i.e., not sending data to BIG-IQ or another external device).
-- There is a large amount of statistics data.
-- May occur even if AVR is not explicitly provisioned.
Impact:
High IO can impact various processes on BIG-IP systems. Some of them can experience timeouts and might restart.
Workaround:
The most effective workaround is to lower the amount of data collected by setting the 'avr.stats.internal.maxentitiespertable' DB variable to a lower value. The recommended values are 20000 (on larger, more powerful systems with more than 16 cores) or 2148 (on smaller systems).
Note: After you lower the database value, continue to monitor the BIG-IP system for long I/O wait times and high CPU usage. If symptoms persist and the system continues to experience resource issues, you may need to reset the BIG-IP AVR statistics. For information about resetting BIG-IP AVR statistics, refer to K14956: Resetting BIG-IP AVR statistics :: https://support.f5.com/csp/article/K14956.
Component: TMOS
Symptoms:
In the deployment of BIG-IP 15.0.0 on Nutanix AHV Hypervisor, f5-label service is failing with inappropriate input device error.
Conditions:
Deployment of BIG-IP v15.0.0 on Nutanix AHV Hypervisor.
Impact:
Deployment of BIG-IP v15.0.0 is not stable to log into GUI or terminal on Nutanix AHV Hypervisor.
Workaround:
Steps:
1. Mount the drive:
mount -o rw,remount /usr
2. Add a comment below the line in the '/usr/lib/systemd/system/f5-label.service' service file:
#StandardInput=tty
3. Reload the daemon:
systemctl daemon-reload
4. Restart the service:
systemctl restart f5-label
Component: TMOS
Symptoms:
Certificate Subject Alternative Names are limited to 1023 characters.
Conditions:
Using a certificate with a Subject Alternative Name longer than 1023 characters.
Impact:
A certificate's Subject Alternative Name is not correct in the BIG-IP configuration.
This does not impact the BIG-IP system's ability to select the proper Client SSL profile on a virtual server that uses SNI matching to provide distinct certificates.
Workaround:
Specify fewer than 1023 character for the Certificate Subject Alternative Names.
Component: Local Traffic Manager
Symptoms:
The IPv6 Management route has lower metrics than the static IPv6 tmm route. As a result, traffic that matches the default route goes to the management interface.
Conditions:
-- Create an IPv6 management route, which is going to be a default gateway.
-- Receive another default gateway from a configured peer using any of dynamic routing protocols (BGP, OSPF, etc.)
Impact:
The incorrect routing table sends the traffic that matches the default route to the management interface.
Workaround:
None.
Component: TMOS
Symptoms:
IPv6 route pool uses an incorrect source address rather than the self IP address. As a side symptom, if there are an even number of members in the pool, only half of the pool members are attempted during load balancing.
Conditions:
--IPv6 route pool is configured.
Impact:
Failed connections from the BIG-IP host that uses an IPv6 pool route.
Workaround:
None.
Component: TMOS
Symptoms:
Many SNMPv3 clients pay attention to the engineBoots value as part of server authentication. When the BIG-IP system is upgraded, the engineBoots value is not retained, so it restarts at 1.
Conditions:
Upgrading a BIG-IP system whose engineBoots value is greater than 1.
Impact:
The engineBoots value is reset to 1. This may look like an error condition for the SNMPv3 client.
Workaround:
1. Run the following command (where n = the value at which you want to start the engineBoots):
tmsh modify sys snmp include 'engineBoots n'
2. Restart SNMPD.
Component: Service Provider
Symptoms:
The "GENERICMESSAGE::message src" and "GENERICMESSAGE::message dst" iRule commands may not work properly if iRule processing changes to a different TMM. These commands may return an empty string rather than correct data.
Conditions:
-- Using "GENERICMESSAGE::message src" and/or "GENERICMESSAGE::message dst" iRule commands.
-- iRule processing moves from one TMM to another TMM.
Impact:
Incorrect data returned from "GENERICMESSAGE::message src" and "GENERICMESSAGE::message dst" iRule commands.
Component: Local Traffic Manager
Symptoms:
For HTTP iRule runtime validation errors, an ERR_NOT_SUPPORTED error message is appended (with rule name and event) to /var/log/ltm, but the message is not informative about the cause of the validation error:
err tmm1[20445]: 01220001:3: TCL error: /Common/example <HTTP_REQUEST> - ERR_NOT_SUPPORTED (line 1) invoked from within "HTTP::uri".
The system should post a more informative message, in this case:
err tmm[10662]: 01220001:3: TCL error: /Common/example <HTTP_REQUEST> - can't call after responding - ERR_NOT_SUPPORTED (line 1) invoked from within "HTTP::uri"
Conditions:
-- HTTP filter and HTTP iRules are used by a virtual server.
-- An HTTP iRule runtime validation error happens. For example, HTTP::uri is called after HTTP::respond () which is not supported.
Impact:
With no informative error messages, it is difficult to identify the validation error.
Workaround:
There is no workaround at this time.
Component: Service Provider
Symptoms:
In SIP ALG virtual with source-nat-policy assigned, if traffic processed by the virtual server does not match source-nat-policy, or if it matches source-nat-policy that does not have source-translation configured, tmm cores and restarts.
Conditions:
-- SIP ALG virtual server with an assigned source-nat-policy.
-- Traffic does not match the source-nat-policy, or traffic matches a source-nat-policy that has no source-translation configured.
Impact:
Traffic disrupted while tmm restarts.
Workaround:
Configure SIP ALG virtual so that the condition never happens. For example, apply a source attribute to the virtual server that filters out traffic that will not match the source-nat-policy. Never use a source-nat-policy that has no source-translation.
Component: TMOS
Symptoms:
A limited cross-site scripting issue was reported affecting the mod_proxy error page. An attacker could cause the link on the error page to be malformed and instead point to a page of their choice. This would only be exploitable where a server was set up with proxying enabled but was misconfigured in such a way that the Proxy Error page was displayed.
Conditions:
This flaw is only exploitable if Proxy* directives are used in Apache httpd configuration. The following command can be used to search for possible vulnerable configurations:
grep -R '^\s*Proxy' /etc/httpd/
Impact:
An attacker could cause the link on the error page to be malformed and instead point to a page of their choice.
Workaround:
This flaw is only exploitable if Proxy* directives are used in Apache httpd configuration. As a Mitigation/Workaround, exclude Proxy* directives in Apache Httpd configuration.
Component: Local Traffic Manager
Symptoms:
In early server responses, the client does not receive the intended response from the HTTP::respond iRule. The client instead receives an unexpected 500 internal server error.
Conditions:
-- A slow client.
-- early server response with the HTTP::respond iRule.
Impact:
A client does not receive the redirect from the HTTP::respond iRule.
Workaround:
None.
Component: TMOS
Symptoms:
Zombie "systemctl" process, as a child of mcpd.
Conditions:
Reboot of the BIG-IP.
Impact:
Minimal; a single zombie process is created.
Workaround:
To get rid of the process, you can restart mcpd.
Component: Local Traffic Manager
Symptoms:
Inconsistent (out-of-sync) LACP working member state.
Incorrect trunk high availability (HA) score.
Conditions:
LACP updates while blade is going offline.
Impact:
Incorrect high availability (HA) score may prevent the unit from automatically failing over.
Component: Application Visibility and Reporting
Symptoms:
Some of the calculations in the AVR publisher are not performed, and incorrect values are shown in the REST response.
Conditions:
-- Device vector detection and mitigation thresholds are set to 10
-- The attack vector is triggered
Impact:
Wrong values appear in REST reponse
Component: Local Traffic Manager
Symptoms:
Forwarding DataBase (FDB) not updated when a MAC moves among interfaces.
Conditions:
-- Having multiple paths to a MAC in a given configuration.
Impact:
There are duplicate MAC address entries which come from multiple interfaces.
Workaround:
None.
Component: TMOS
Symptoms:
The BIG-IP system does not generate signature files for logs. As a result, the system stores the audit information (i.e., the log files stored in /var/log folder and other subfolders) in an incorrect format.
Conditions:
Viewing the audit information stored in /var/log and other locations.
Impact:
Messages are stored in an incorrect format.
Workaround:
Disable local logging for audit logs and send them to remote syslog, for example:
tmsh modify sys syslog include "filter f_audit { facility(local0) and not message(AUDIT); }; "
Component: TMOS
Symptoms:
TMM asserted and cored in racoon2 with this panic message:
panic: iked/ikev2_child.c:2858: Assertion "Invalid Child SA proposal" failed.
Conditions:
When IKEv2 Phase 2 SA has no peer proposal associated with it.
Impact:
Traffic disrupted while tmm restarts.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
Server syn is flowing on the wrong VLAN, when tmm tries to establish a server connection. The BIG-IP system sends RST packets of unmatched VLAN/MAC combination
Conditions:
-- Virtual wire is configured .
-- Clientside data and handshake come in on different VLANs.
Impact:
Some client connections fail to establish
Workaround:
None.
Component: TMOS
Symptoms:
The session and authentication cookies are created using a limited character set.
Conditions:
Creating session and authentication cookies.
Impact:
Cookies are created with a less broad character set than they could be.
Workaround:
None.
Behavior Change:
This release changes the format of the BIGIPAuthCookie and JSESSIONID cookies to use a larger alphabet during encoding (case sensitive alphanumeric).
Component: TMOS
Symptoms:
You see duplicate log messages in /var/log/kern.log
Conditions:
This can be encountered when viewing /var/log/kern.log right after startup in BIG-IP versions dating back to 14.1.0
Impact:
Viewing ('cat'ing) kern.log results in duplicated log messages in the buffer.
Component: Service Provider
Symptoms:
When processing a GTP message containing zero-length IEs (which are allowed by the 3GPP Technical Specification), the message might get rejected.
Conditions:
Virtual server with GTP profile enabled processing GTP traffic.
Impact:
Well-formed GTP messages might get rejected.
Workaround:
Avoid sending GTP messages containing zero-length IEs.
Component: TMOS
Symptoms:
There is no option to configure a PPTP profile for a virtual server in the GUI.
Conditions:
Creating or modifying a virtual server in the GUI.
Impact:
Unable to configure the PPTP profile for a virtual server using the GUI.
Workaround:
Use TMSH to add a PPTP profile to the virtual server.
Component: Local Traffic Manager
Symptoms:
Missing virtual server name in Hardware Syncookie activation logs. ltm/logs contains error messages:
notice tmm2[1150]: 01010240:5: Syncookie HW mode activated, server = 0.0.0.0:0, HSB modId = 2.
Conditions:
-- More than one virtual server with same Destination IP e.g., 'x.x.x.x'.
-- Port 'y' configured.
-- Hardware Syncookie activated.
Impact:
Difficult to determine which virtual server actually got the Syncookie activated.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
In certain configurations, and when the BIG-IP system does not have a directly connected route to the request sender, proxy ARP may fail, leading to dropped ARP replies.
Conditions:
-- Running v12.1.4.1 or 12.1.3.7 with engineering hotfix 0.89.2.
-- ARP requests and replies are processed by different TMMs.
-- A directly connected route to the request sender is not available.
Impact:
ARP replies are dropped, leading to connection failures.
Workaround:
Create a self IP in the same subnet as the ARP request senders. This creates the necessary directly connected route.
Component: TMOS
Symptoms:
MCPD memory consumption may increase on standby BIG-IP device if APM configuration is updated. Some of the allocated memory is not freed after configuration update.
Conditions:
-- BIG-IP high availability (HA) pair is installed and configured
-- APM is provisioned
-- Access Policy is configured and updated periodically
Impact:
MCPD may take a lot of memory on the standby device. Normal functionality of standby device may be stopped; reboot of the device is required.
Component: TMOS
Symptoms:
The engineID, engineBoots, engineTime values in SNMPv3 traps are shared by both the SNMP and the Alert daemons and are included in traps raised by both daemons. When the engineID is reconfigured then both daemons must be restarted in order to resynchronize the new values.
Conditions:
Traps issued by the SNMP and Alert daemons may not have engine values that are in sync when the EngineID is first reconfigured. This can happen both with a configuration change and an upgrade.
Impact:
This may confuse the SNMP client receiving the trap.
Workaround:
Restart the snmp daemon and then the alert daemon when the engine ID is reconfigured for the first time and the first time after a software upgrade
tmsh restart sys service snmpd alertd
Component: TMOS
Symptoms:
Packets are being sent to the AWS instance but no packets are seen on interface.
Conditions:
-- AWS instance using xnet driver.
-- Occurs when the instances are idle and then suddenly passes traffic again.
-- Other, more specific conditions are unknown at this time.
Impact:
Loss of packets in the interface, in turn, causing data loss.
Workaround:
A temporary way to avoid the problem is to configure BIG-IP Virtual Edition (VE) to use an alternative network driver in place of the default 'xnet' driver. In releases 14.1.0 and later, this would be the 'sock' driver.
Use The following command sequences from the BIG-IP instance's 'bash' prompt to configure the alternative driver. (Note the use of the 'greater-than' symbol.)
# echo "device driver vendor_dev 1d0f:ec20 sock" > /config/tmm_init.tcl
[check that the file's contents are correct]
# cat /config/tmm_init.tcl
[restart the BIG-IP's TMM processes]
# bigstart restart tmm
[make certain that the 'driver_in_use' is 'sock']
# tmctl -dblade -i tmm/device_probed
Component: TMOS
Symptoms:
Attempts to login to the serial console with remote user credentials (e.g., RADIUS, LDAP, TACACS remote auth) fail with one of the following error messages:
-- 'Cannot load user credentials for user' (v13.1.1.2)
-- 'Session setup problem, abort.' (v14.1.0.1)
Conditions:
Configure system for remote authentication and attempt authentication via serial console.
Impact:
Remote authentication users are unable to login via serial console.
Workaround:
There are two workarounds:
-- Remote authentication users can login using an SSH connection to the BIG-IP system's management IP address.
-- Use the credentials of a local user account to login to the serial console.
Component: TMOS
Symptoms:
In rare circumstances, when a reboot immediately follows a VIPRION blade failover, a REBOOT REQUIRED prompt will appear on one blade after the system starts up again.
Conditions:
This issue can be created by doing the following:
- using a VIPRION system with at least 2 blades running
- AAM is not provisioned
- reset the primary blade
- immediately following the blade reset, run 'clsh reboot' on a secondary blade.
Impact:
Following the clsh reboot, the REBOOT REQUIRED prompt appears on one blade:
[root@vip4480-r44-s18:/S2-yellow-S::REBOOT REQUIRED:Standalone] config #
Any blade with this prompt must be rebooted again.
Workaround:
None currently known.
Component: TMOS
Symptoms:
System logs error:
err sod[8444]: 01140003:3: Out of shmem, increment amount in /etc/ha_table/ha_table.conf.
Conditions:
-- Large number of traffic groups.
-- A number of devices in the device cluster.
-- Heavy traffic resulting in numerous configsync or config save operations.
Impact:
Memory leak. Future changes to the high availability (HA) table may fail or be ignored. This could result in HA events not being tracked correctly.
Workaround:
None.
Component: TMOS
Symptoms:
The Management interface flaps after rebooting the device, which may cause a momentary active-active condition in a high availability (HA) configuration.
Conditions:
This can occur after rebooting the active or standby device in an HA configuration if the final management port configuration completes late in the startup sequence. This can be due to network conditions for the network the management port is connected to.
This problem has been observed only on hardware platforms.
Impact:
Devices go active-active for a few seconds and then resume normal operation.
Workaround:
You may be able to work around this by changing the management port speed to 100/Fixed Duplex.
For more information on changing the interface, see K14107: Configuring the media speed and duplex settings for network interfaces (11.x - 13.x), available at https://support.f5.com/csp/article/K14107.
Component: TMOS
Symptoms:
If the SNMP server receives an SNMPv3 request with a small max message size then, it applies that check to all requests. This can cause SNMPv1 and SNMPv2c requests time out if they are too long or if their responses are too long, for example, large get bulk requests.
Conditions:
An SNMPv3 small max message size received while processing large SNMPv1 and SNMPv2c requests.
Impact:
Responses time out.
Workaround:
Do not send SNMPv3 requests to the BIG-IP system.
Component: Local Traffic Manager
Symptoms:
In BIG-IP GUI iRule definitions, when hovering over HTTP::proxy, the help text mentions 'HTTP::proxy dest', which is an invalid command.
Conditions:
The system displays incorrect information when the iRule help text is visible.
Impact:
The help text mentions 'HTTP::proxy dest', which is an invalid command option.
Workaround:
Do not use the invalid 'HTTP::proxy dest' command.
Component: Local Traffic Manager
Symptoms:
When multiple users are issuing REST commands, memory may leak slowly in icrd_child.
Conditions:
-- The icrd_child process is running.
-- There are multiple users accessing device via REST.
Impact:
The memory leak is very progressive. Eventually, the icrd_child process runs out of memory.
Workaround:
None.
Component: Application Security Manager
Symptoms:
A CSRF false-positive violation.
Conditions:
CSRF enforcing security policy.
This is a very rare scenario, but it happens due to a specific parameter in the request, so the false-positive might repeat itself many times for the same configuration.
Impact:
False-positive Blocking / Violation
Workaround:
If this happens change the csrf parameter and restart the asm daemon:
1. Change the csrf parameter name internal parameter:
/usr/share/ts/bin/add_del_internal add csrf_token_name <string different than csrt>
2. Restart the asm daemon:
restart asm
Component: Local Traffic Manager
Symptoms:
ID746077 changed the dhcprelay behavior in order to comply with RFC 1542 Clarifications and Extensions for BOOTP.
However, as the change also encompasses the DHCP-to-DHCP relay scope, the behavior cannot be configurable with a db key.
Conditions:
DHCP Relay deployments where the giaddr needs to be changed.
Impact:
You are unable to specify whether giaddr will be changed.
Workaround:
None.
Component: TMOS
Symptoms:
When a transaction attempts multiple commands (delete, create, modify) for the same object in the same transaction, the results can be unexpected or undefined. A common example is: 'transaction { delete key create_if key }' where the transaction attempts 'delete key', and then 'create_if key', which unmarks the delete operation on the key (so in this case the key remains unmodified). In other cases it is possible that monitoring stops for the associated object, such as for: pool, pool_member, node_address, monitor.
Conditions:
A user-initiated transaction attempts multiple commands for the same monitor-related object (such as delete, create, modify).
Impact:
The GUI shows misleading info about the pool monitor.The monitor-related object may be unchanged, or monitoring may stop for that object.
Workaround:
Transactions modifying a monitor-related object (pool, pool_member, node_address, monitor) should perform a single command upon that object (such as one of: 'delete', 'create', 'modify').
Component: TMOS
Symptoms:
In device group has enabled 'save sys config' for all auto-sync operations using the following command:
modify cm device-group name save-on-auto-sync true
Warning: Enabling the save-on-auto-sync option can unexpectedly impact system performance when the BIG-IP system automatically saves a large configuration change to each device.
Conditions:
-- The save-on-auto-sync option is enabled.
-- Device has large configuration, such as 2,100 virtual servers and ~1100 partitions
Impact:
Configuration is not saved, which leads to out-of-sync condition.
Workaround:
You can avoid this issue by using manual sync instead of auto-sync, or by not enabling 'save-on-auto-sync'.
Component: TMOS
Symptoms:
Beginning in version 14.1.0, LDAP debugging must be performed on nslcd logs and not pam_ldap logs; however, it is not possible to enable debug logging on nslcd via the configuration file.
Conditions:
This would be encountered only if you (or F5 Support) wanted to do troubleshooting of LDAP connections by enabling debug logging.
Impact:
LDAP system authentication 'debug' parameter does not provide sufficient levels of debug logs, but there is no functional impact to normal system operation.
Workaround:
To enable debug logging and have the system post log messages to the SSH/console window, start the nslcd process with -d option, which causes nslcd to run in the foreground until you press control-c to stop it:
systemctl stop nslcd
nslcd -d
Note: The -d setting does not persist, so each time you want to log debug output, you must complete this procedure.
You can increase the amount of debug output by specifying additional -d options (up to 3), e.g., '-ddd' or '-d -d -d'.
When done, stop nslcd with control-c, and then restart it with the default options via the normal systemctl daemon:
systemctl start nslcd
Component: TMOS
Symptoms:
Running the command 'tmsh list' on a pool or virtual server with the 'all-properties' argument generates a warning:
[api-status-warning] ltm/virtual, properties : deprecated : urldb-feed-policy
Conditions:
Including the 'all-properties' argument with the 'tmsh list' command.
Impact:
There is no impact to the system. The excessive [api-status-warning] at stderr and /var/log/ltm for tmsh list commands are spurious, benign, and can be ignored.
Workaround:
tmsh modify /mgmt shared settings api-status log resource-property deprecatedApiAllowed false
tmsh modify /mgmt shared settings api-status log resource deprecatedApiAllowed false
Component: Access Policy Manager
Symptoms:
WebSSO crashes.
Conditions:
When the optional field of the target info is about 1000 bytes or larger.
Impact:
WebSSO crashes and loss of service.
Workaround:
Config NTLM not to have large target info, recommend < 800.
Component: Service Provider
Symptoms:
When MRF SIP is configured in per-client mode and preserve-strict source port is enabled on a virtual server, messages may fail to forward due to port collisions when multiple clients try to use the same port (which is expected/accepted behavior with this configuration). After the port has been freed or the configuration changed, messages continue to fail for clients that had previous port collisions.
Conditions:
-- MRF SIP configured with: Per-Client connection mode and virtual server with preserve-strict source port enabled.
-- Multiple clients try to connect using the same local port.
-- Previously failed client connections attempt to connect again after the port has been freed or configuration changed.
Impact:
Calls from one or more clients are unable to be completed.
Workaround:
You can prevent this behavior using either workaround:
-- Configure a different connection mode (Per-TMM, for example).
-- Disable preserve-strict source port on the virtual server.
Component: Local Traffic Manager
Symptoms:
Intermittently (depending the timing of operations that keep MCP busy):
1. Messages similar to the following may be logged in the LTM log, indicating that the virtual server associated with a pool became temporarily unavailable:
-- notice mcpd[4815]: 01071682:5: SNMP_TRAP: Virtual /Common/test_vs has become unavailable.
-- notice mcpd[4815]: 01071681:5: SNMP_TRAP: Virtual /Common/test_vs has become available.
2. Optionally, if a 'min-up-members' value is configured for the pool, a message similar to the following may be logged in the LTM log, indicating that the number of available pool members became less than the configured value:
-- notice mcpd[4815]: 01070282:3: Number of pool members 2 less than min up members 3.
Conditions:
1. The pool members are all FQDN pool members.
2. The DNS query to resolve pool member FQDNs returns a completely new (non-overlapping) set of IP addresses.
(This causes all existing Ephemeral pool members to be removed and replaced with new Ephemeral pool members.)
3. MCP is very busy and slow to process messages.
Impact:
Under these conditions, existing Ephemeral pool members may be removed before new Ephemeral pool members can be created to replace them, causing the pool member to become temporarily empty. This can result in intermittent loss of availability of the virtual server if all records returned by the DNS server for the referenced FQDN change from the previous response.
Workaround:
None.
Component: TMOS
Symptoms:
When creating 'performance HTTP' virtual servers via GUI, the following error is reported:
01070734:3: Configuration error: A Virtual Server(/Common/vfasthttp) cannot be associated with both fasthttp and L4 profile.
Conditions:
Use the GUI to create a virtual server of type Performance (HTTP).
Impact:
Failed to create a 'performance HTTP' virtual server.
Workaround:
Use TMSH to configure the performance HTTP virtual server:
tmsh create ltm virtual vfasthttp destination 1.1.1.1:80 ip-protocol tcp profiles add { fasthttp }
Component: Advanced Firewall Manager
Symptoms:
The /var partition might become completely full on the disk due to files being written to /var/config/rest. This condition may be accompanied by console error messages similar to the following:
011d0004:3: Disk partition /var (slot #) has only 0% free on secondary blade.
Additionally, there may be periodic restjavad and bigd daemons restarts related to disk space exhaustion.
Conditions:
Process traffic while DoS Dashboard is open
Impact:
The partition housing /var/config/rest may become 100% full, impacting future disk IO to the partition.
Workaround:
Important: This workaround is temporary, and may need to be periodically performed either manually or from a script.
Impact of Workaround: While these steps are performed, the BIG-IP REST API will be temporarily inaccessible, and higher disk IO may be seen.
Run the following commands, in sequence:
bigstart stop restjavad
rm -rf /var/config/rest/storage*.zip
rm -rf /var/config/rest/*.tmp
bigstart start restjavad
Manual application of these workaround steps clears the 100% utilized space condition and allows the partition to resume normal operation.
Component: TMOS
Symptoms:
Gossip appears Active even when one or more devices go missing from device group. 'restcurl shared/gossip' shows active on both devices, even when the devices are not listed in 'restcurl shared/resolver/device-groups/tm-shared-allBIG-IPs/devices'.
Conditions:
The conditions under which this issue occurs are unknown. This is an intermittent issue.
Impact:
Gossip reports that it is working when it is not.
Workaround:
-- If the missing device is the active device, run the following command on the Active DSC Device:
restcurl -X POST -d '{}' tm/shared/bigip-failover-state
-- If the missing device is the standby device, reboot the device, make it active, and then run the following command:
restcurl -X POST -d '{}' tm/shared/bigip-failover-state
Component: Local Traffic Manager
Symptoms:
A virtual-wire that uses interface trunks may use a single interface on egress.
Conditions:
Virtual-wire configured across multi-interface trunks.
Impact:
This may lead to unexpected link saturation.
Workaround:
None.
Component: Application Security Manager
Symptoms:
ASM logrotate reports errors in /var/log/asm.:
error: error creating output file /ts/log//bd.log.1: File exists
Conditions:
Files ending with .1 exists in the logs directories.
Impact:
Logrotate does not work. May fill disk with logs over time.
Workaround:
Remove or rename all of the .1 logs.
Component: TMOS
Symptoms:
The BIG-IP system may fail to deploy new or reconfigure existing iApps. When this happens, a long error message is displayed in the GUI that begins with:
script did not successfully complete: ('source-addr' unexpected argument while executing
The message is also logged to /var/log/audit by scriptd with a severity of 'notice'.
The unexpected argument mentioned in the error varies depending on the iApp being deployed and on the settings you configure. You may also see 'snatpool', 'ldap', etc.
Conditions:
This issue occurs when:
-- The BIG-IP system is configured with multiple users of varying roles.
-- The scriptd daemon has already spawned the maximum number (5) of allowed child processes to serve its queue, and all the processes were assigned a low 'security context'. This can happen, for instance, if a low-privileged user (such as an Auditor) has been looking at the configuration of iApps using the GUI a lot.
-- Subsequently, a high-privileged user (such as an Administrator) attempts to deploy a new iApp or reconfigure an existing one.
Note: You can inspect the number of child processes already created by scriptd by running the following command:
pstree -a -p -l | grep scriptd | grep -v grep
However, it is not possible to determine their current 'security context'.
Impact:
New iApps cannot be deployed. Existing iApps cannot be re-configured.
Workaround:
Restart scriptd. To restart scriptd, run:
bigstart restart scriptd
Running this command has no negative impact on the system.
The workaround is not permanent; the issue may occasionally recur depending on your system usage.
Component: TMOS
Symptoms:
Errdefsd crashes, creates a core file, and restarts.
Conditions:
The conditions under which this occurs are unknown.
Impact:
Possible loss of some logged messages.
Workaround:
None.
Component: TMOS
Symptoms:
When loading a UCS file from one iSeries model to a different iSeries model, the FPGA fails to load due to a symlink in the UCS file pointing to the firmware version for the source device.
The system will remain in INOPERATIVE state, and messages similar to the following will be seen repeatedly in /var/log/ltm:
-- emerg chmand[7806]: 012a0000:0: FPGA firmware mismatch - auto update, No Interruption!
-- emerg chmand[7806]: 012a0000:0: No HSBe2_v4 PCIs found yet. possible restart to recover Dataplane.
-- emerg chmand[7806]: 012a0000:0: Dataplane INOPERABLE - Incorrect number of HSBs:0, Exp:1, TMMs: 2
-- err chmand[7806]: 012a0003:3: HAL exception publishing switch config: Dataplane INOPERABLE - Incorrect number of HSBs:0, Exp:1, TMMs: 2
Conditions:
Loading a UCS from one iSeries model onto another model, for example, from an i7800 onto an i11400-ds, or from an i2600 to an i5600.
Impact:
FPGA fails to load; the BIG-IP system becomes unusable.
Workaround:
1. Update the symbolic link /config/firmware/hsb/current_version to point to the correct firmware file for the hardware model in use. Here are some examples:
-- For the i2800:
# ln -sf /usr/firmware/hsbe2v4_atlantis/L7L4_BALANCED_FPGA /config/firmware/hsb/current_version
-- For the i7800:
# ln -sf /usr/firmware/hsbe2v2_discovery/L7L4_BALANCED_FPGA /config/firmware/hsb/current_version
-- For the i11400-ds:
# ln -sf /usr/firmware/hsbe2_discovery_turbo/L7L4_BALANCED_FPGA /config/firmware/hsb/current_version
2. Reboot the system
Component: TMOS
Symptoms:
A vCMP guest cannot access software images and hotfix ISOs from the host. The vCMP host cannot gather status information from the vCMP guest, for example, high availability (HA) status, provisioning, and installed software information.
Conditions:
The TMUI redirect-http-to-https is enabled.
Impact:
A vCMP guest cannot access software images and hotfix ISOs from the host. The vCMP host cannot gather status information from the vCMP guest, for example, HA status, provisioning, and installed software information.
Workaround:
On the vCMP guest, disable sys httpd redirect-http-to-https.
Component: Application Security Manager
Symptoms:
If an Attack Signature update encounters an error during installation, files that are meant to be temporary are left behind on disk and a not subject to a periodic cleanup. This can eventually lead to disk space issues.
Conditions:
Attack Signature update encounters an error during installation.
Impact:
This can eventually lead to disk space issues.
Workaround:
Old sigfile.tmp.* directories under /var/ts/var/tmp can be safely removed.
Component: Service Provider
Symptoms:
In MRF/Diameter deployment, the LTM pool 'Current Sessions' statistics may show an unusually large number, such as 18446744073709551606.
Conditions:
There is a Diameter answer that does not match a pending request, the answer message is dropped, but BIG-IP system still decrements the 'Current Sessions' counter. If the counter is already zero, it can underflow.
Impact:
'Current Sessions' statistics can be used to track number of pending requests in the queue. When it underflows, the number becomes useless, making troubleshooting more difficult.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
An In-TMM monitor may be marked down when the CMP-hash (Cluster Multiprocessing) is set to non-default value.
Conditions:
-- There is a configured In-TMM monitor (K11323537).
-- CMP-hash is set to non-default value.
Note: For information about In-TMM monitoring, see K11323537: Configuring In-TMM monitoring :: https://support.f5.com/csp/article/K11323537.
Impact:
An In-TMM monitor is falsely marked as down.
Workaround:
Use default settings for a CMP-hash.
Component: Advanced Firewall Manager
Symptoms:
'TCP half open' reports drops in logs/tmctl/AVR even though it is configured in detect-only mode.
Conditions:
-- 'TCP half open' attack is being actively detected.
-- LTM syn-cookie mitigation is enabled.
-- This is triggered when LTM syn-cookies mitigation begins.
Impact:
It will appear that 'TCP half open' is doing mitigation, but it is actually LTM syn-cookies dropping the connections.
Workaround:
If LTM syn-cookies are not needed, disable the option:
modify ltm global-settings connection default-vs-syn-challenge-threshold infinite global-syn-challenge-threshold infinite
Component: Global Traffic Manager (DNS)
Symptoms:
Users logged in with a role of resource-administrator are unable to create a GTM server object via GUI. The warning banner reports 'No Access'.
Conditions:
A user with a role of resource-administrator attempts to create a GTM server object.
Impact:
Unable to create GTM server object via the GUI.
Workaround:
Use tmsh or iControl/REST.
Component: TMOS
Symptoms:
The Link Status traps, both F5 proprietary and standard LinkUp/LinkDown are issued on the BIG-IP hardware but not on BIG-IP Virtual Edition (VE) configurations.
Conditions:
This occurs when interfaces on hardware-based BIG-IP systems or VE-based BIG-IP configurations experience link status events (links go up or down, or are administratively enabled or disabled).
Impact:
Log messages are issued and SNMP traps are issued if an SNMP trap destination is configured.
On a VE-based BIG-IP system, these logs and traps do not occur.
An SNMP client waiting for a Link Status trap on an administrative enable or disable then, does not receive the trap.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
Pool state shows down when there is no route configured to node.
Conditions:
-- In-tmm gateway_icmp monitor configured for a node or pool member.
-- There is no route to the node or pool member.
Impact:
The pool member or node is marked down and the reason listed is 'timeout', instead of 'no route to host'.
Workaround:
None.
Component: TMOS
Symptoms:
A CMP (Clustered Multiprocessing) state change occurs when the state of the BIG-IP system changes.
This happens in the following instances:
- Blade reset.
- Booting up or shutting down.
- Running 'bigstart restart'.
- Setting a blade state from/to primary/secondary.
During these events, Bidirectional Forwarding Detection (BFD) session processing ownership might be migrating from old, processing TMMs to new, selected TMMs. This process is rapid and could lead to contest between several TMMs over who should be the next BFD processing owner.
It might also lead to a situation where the BFD session is deleted and immediately recreated.
This problem occurs rarely and only on a chassis with more than one blade.
Conditions:
-- VIPRION chassis with more than one blade.
-- CMP hash of affected VLAN is changed from the Default value, for example, to Source Address.
-- BFD peering is configured.
-- CMP state change is occurred on one of the blades.
-- BFD connection is redistributed to the processing group (TMMs) on the blade that experienced the CMP state change and the contest between the old TMM owner and the new TMM owner occurs.
Impact:
When the BFD session is recreated, it marks corresponding routing protocol DOWN if it's configured. The protocol might be BGP, OSPF, or any other routing protocols that support BFD.
This causes the routing protocol to withdraw dynamic routes learnt by the configured protocol, making it impossible to advertise dynamic routes of affected routing protocols from the BIG-IP system to the configured peers. This can lead to unexpected routing decisions on the BIG-IP system or other devices in the routing mesh.
In most cases, unexpected routing decision are from networks learnt by affected routing protocols when the routing process on the BIG-IP system become unreachable. However, this state is short-lived, because the peering will be recreated shortly after the routing protocol restarts. The peering time depends on the routing configuration and responsiveness of other routing devices connected to the BIG-IP system. It's the usual routing convergence period, which includes setting the peering and exchanging routing information and routes.
Workaround:
There are two workarounds, although the latter is probably impractical:
-- Change CMP hash of affected VLAN to the Default value.
-- Maintain a chassis with a single blade only. Disable or shut down all blades except one.
Component: Service Provider
Symptoms:
A configuration warning is produced when the RADIUS avp command is used with a variable instead of a constant, for example:
warning: [The following errors were not caught before. Please correct the script in order to avoid future disruption. "unexpected end of arguments;expected argument spec:integer"102 45][RADIUS::avp replace USER-NAME $custom_name]
This appears to be benign, as the configuration loads successfully, and the script works as expected.
Conditions:
Using:
RADIUS::avp replace USER-NAME $custom_name
Instead of:
RADIUS::avp replace USER-NAME "static value"
Impact:
Incorrect warning in log. You can ignore these messages, as the configuration loads successfully, and the script works as expected.
Workaround:
This warning is benign, as the configuration loads successfully, and script works as expected.
Component: Global Traffic Manager (DNS)
Symptoms:
DNS cache idx conflicts and tmm crash.
Conditions:
-- High availability (HA) configuration.
-- DNS cache is configured and synced to the peer twice
-- A second DNS cache is configured on the peer.
Impact:
The idx conflicts will be observed. If the second DNS cache is of another type and is added to a virtual server, accessing that virtual server might cause a tmm core. Traffic disrupted while tmm restarts.
Workaround:
On the BIG-IP system that has the DNS cache idx conflicts, restart tmm:
# bigstart restart tmm
Component: Local Traffic Manager
Symptoms:
- Monitors are firing and are being sent to a pool-member or node address rather than a monitor's alias address.
- Running the command 'tmsh load /sys config' reports an error:
01070038:3: Monitor /Common/a-tcp address type requires a port.
Conditions:
-- Create a monitor without an alias address.
-- Modify the monitor later in the TMUI to specify an alias address.
Impact:
Monitors are sent to an incorrect IP address.
tmsh load /sys config will fail to load the configuration.
Workaround:
There are two workarounds:
-- Delete and recreate the monitor and specify the correct alias address at creation time.
-- Fix the monitor definition using tmsh.
Component: TMOS
Symptoms:
Unable to login as remote-user.
Conditions:
When the following are true:
-- LDAP remote-auth configured with user-template.
-- Remote-user configured to permit login.
Impact:
Unable to login with remote-user.
Workaround:
Use bind-dn for authentication.
Component: Application Security Manager
Symptoms:
When creating a whitelist in the Bot Defense profile with url "Any" - after modifying the Bot Defense log profile, the whitelist does not apply anymore.
Conditions:
-- Bot Defense profile is attached to the Virtual Server
-- Adding a whitelist to the Bot Defense profile with url "Any"
-- Modifying the Bot Defense profile afterwards.
Impact:
Whitelist does not apply - users from the defined IP/GEO location might be blocked.
Workaround:
Delete and add the whitelist after modifying the profile.
Component: Global Traffic Manager (DNS)
Symptoms:
The TMM may crash and restart when an iRule on a DNS virtual server performs the 'drop' command while the BIG-IP system is handling both a DNS request and DNS response at the same time for the same DNS client IP and port without UDP Datagram-LB.
Conditions:
-- The BIG-IP instance has two or more TMM processes as a result of having two or more physical cores or virtual CPUs.
-- A virtual server with both DNS and UDP profiles and one or more iRules.
-- The UDP profile has Datagram LB disabled.
-- The iRules have a 'drop' command.
-- The iRules have a DNS_REQUEST and/or DNS_RESPONSE event with an iRule command that require coordinating data with another TMM on the system, such as the 'table' command.
Impact:
TMM crash or restart. Traffic impacted. Traffic disrupted while tmm restarts.
Workaround:
F5 strongly recommends using a UDP profile with Datagram-LB enabled for DNS UDP virtual servers.
Alternatively, replace the 'drop' command with DNS::drop in DNS_REQUEST and DNS_RESPONSE events, or with UDP::drop in other iRule events.
See the respective references pages for DNS::drop and UDP::drop for the Valid Events each iRule command is available in:
https://clouddocs.f5.com/api/irules/DNS__drop.html
https://clouddocs.f5.com/api/irules/UDP__drop.html
Component: TMOS
Symptoms:
While installing a new version in another partition, the GUI displays an error for a brief time:
'Install Status':Failed Troubleshooting
Conditions:
Install a new version in another partition.
Impact:
The GUI error is misleading. It is showing the install status as 'Failed Troubleshooting' even though the installation is proceeding normally. The installation process is proceeding normally; only the error is incorrect and does not indicate a problem with the installation.
Workaround:
If you click on the 'Troubleshooting' link on the GUI screen, the GUI indicates that it is actually installing properly without any error.
Component: Application Security Manager
Symptoms:
CPB Connection (between BIG-IP and BIG-IQ logging node) is not refreshed to use the new certificate / new domain name to validate the certificate.
Conditions:
Either:
-- BIG-IQ logging node domain name updated.
-- BIG-IQ logging node webd certificate is replaced (and updated using webd restart).
Impact:
CPB Connection (between BIG-IP and BIG-IQ logging node) remains the same and is not refreshed to use the new certificate.
Workaround:
Restart Policy Builder on the BIG-IP system:
killall -s SIGHUP pabnagd
Component: TMOS
Symptoms:
Some values sent to SNMP can grow too large over time, causing an integer truncation error.
Conditions:
Values greater than 32 bits sent to SNMP.
Impact:
SNMP values are truncated. An error message is logged in var/log/daemon.log:
err snmpd[20680]: truncating integer value > 32 bits
Workaround:
No current workaround.
Component: Global Traffic Manager (DNS)
Symptoms:
Tmsh show gtm pool and show gtm wideip commands with field-fmt will display the object type twice in the output. For example:
tmsh> show gtm pool a field-fmt
gtm pool pool emptypool:A
tmsh> show gtm wideip a field-fmt
gtm wideip wideip testwip.f5.com:A
Conditions:
This occurs when running the following tmsh commands:
tmsh show gtm pool <poolname> field-fmt
tmsh show gtm wideip <wideipname> field-fmt
Impact:
The output type is printed twice
Workaround:
None.
Component: Access Policy Manager
Symptoms:
If the device is active at the time and is passing traffic, if the SSL Orchestrator configuration is deleted, tmm can core.
Conditions:
SSL Orchestrator device is active and passing traffic while being deleted.
Impact:
TMM cores. Traffic disrupted while tmm restarts.
Workaround:
None.
Component: TMOS
Symptoms:
Linux kernel NFS41+ subsystem use-after-free vulnerability when node have NFSv41+ mounts inside several net namespaces.
Conditions:
NFS41+ shares mounted in different network namespaces at the same time can make bc_svc_process() use wrong back-channel IDs and cause a use-after-free vulnerability. Thus a malicious container user can cause a host kernel memory corruption and a system panic.
Impact:
BIG-IP is not exposed to this vulnerability. use-after-free causes system panic, subsequent system reset.
Workaround:
None.
Component: Access Policy Manager
Symptoms:
Tmsh accepts long localdb instance names, but ldbutil later refuses to work with names longer than 64 characters.
The GUI limits the instance name length to 64 characters including the partition prefix, but this is not obvious to the admin.
Conditions:
-- Create a 64 character long local user database instance using tmsh.
-- Try to add users to this instance or try to delete the instance from the GUI.
Impact:
A tmsh-created localdb instance with a name length greater than 64 characters can be created but cannot be used.
Workaround:
Delete instance from tmsh and re-create it with a shorter name.
Component: Access Policy Manager
Symptoms:
Upgrade of standby BIG-IP system results in empty Local Database.
Conditions:
This happens on standby device in a high availability (HA) setup.
Impact:
All previously existing local users disappear from the standby device. If a failover happens, then none of the local users will be able to login now.
Workaround:
To trigger a full database dump from the active BIG-IP system that returns the standby device's database to its original state, on the standby device, after rebooting into the volume with the upgraded installation, do the following:
1. Force stop the localdbmgr process:
bigstart stop localdbmgr
2. Wait at least 15 minutes.
3. Restart the localdbmgr:
bigstart restart localdbmgr
Component: TMOS
Symptoms:
Unable to renew Device Certificate from System :: Certificate Management : Device Certificate Management : Device Certificate :: server.crt in non-English BIG-IP configurations.
Conditions:
Attempting to renew a device certificate on the System :: Certificate Management : Device Certificate Management : Device Certificate :: using the server.crt-equivalent on a non-English BIG-IP system.
Impact:
Unable to renew Device Certificate from the BIG-IP Configuration Utility.
Workaround:
Use a command of the following syntax, replacing key name, cert name, and # of days with your values:
openssl req -new -x509 -key ../ssl.key/server.key -days <# of days> -out server.crt
For example, to renew the siteserver.key and siteserver.crt for 90 days, use the following command:
openssl req -new -x509 -key ../ssl.key/siteserver.key -days 90 -out siteserver.crt
Component: Global Traffic Manager (DNS)
Symptoms:
During routine iQuery SSL renegotiation, the iQuery connection will occasionally be reset.
Conditions:
This occurs occasionally during routine renegotiation.
Impact:
The affected iQuery connection is briefly marked down as the connection is marked down before the connection is immediately re-established.
Workaround:
There is no workaround.
Component: Fraud Protection Services
Symptoms:
The ${geo} pattern in the Logging Profile has only GeoIP data; however, in alerts that are sent to either the alert server or to BIG-IQ, the GEO data includes both GeoIP and GeoLocation information.
Conditions:
Configure ${geo} pattern in Logging Profile template and trigger a logging event.
Impact:
GeoLocation data is missing in logs written by Logging Profile when using ${geo} pattern.
Component: TMOS
Symptoms:
After installing over the top of a previous version, the Management GUI is inaccessible while SSH access works. You may see one or more of the following conditions
Shell prompt may show logger[1234]: Re-starting named
bigstart restart httpd fails
bigstart start httpd fails
Conditions:
Installation over a previously used Boot Volume
Impact:
Corrupt install
Workaround:
Boot back to previous boot volume and then delete the boot volume containing the failed install.
Component: Local Traffic Manager
Symptoms:
The IPv6 Management route has lower metric than the static IPv6 tmm route. As a result, traffic that matches the default route goes to the mgmt interface.
Conditions:
Create an IPv6 mgmt route and a static IPv6 tmm route on the same BIG-IP system. IPv6 routes from TMM are injected at metric 1024.
Impact:
The incorrect routing table sends the traffic that matches the default route to the mgmt interface.
Workaround:
None.
Component: Advanced Firewall Manager
Symptoms:
The system posts the following message when the LDAP authenticated Firewall Manager role creates/modifies a firewall policy with rules or upgrading existing firewall policy:
User does not have modify access to object (fw_uuid_config).
Conditions:
-- Log in using an account with the Firewall Manager role.
-- Create/modify firewall policy with rules or upgrade existing firewall policy.
Impact:
Firewall modification operations fail with access to object (fw_uuid_config) error.
Workaround:
None.
Component: TMOS
Symptoms:
Some logs write the hostname with periods (eg, say for FQDN. For example, /var/log/user.log and /var/log/messages files log just the hostname portion:
-- user.log:Aug 5 17:05:01 bigip1 ).
-- messages:Aug 5 16:57:32 bigip1 notice syslog-ng[2502]: Configuration reload request received, reloading configuration.
Whereas other log files write the full name:
-- daemon.log:Aug 5 16:58:34 bigip1.example.com info systemd[1]: Reloaded System Logger Daemon.
-- maillog:Aug 5 16:55:01 bigip1.example.com err sSMTP[12924]: Unable to connect to "localhost" port 25.
-- secure:Aug 5 17:02:54 bigip1.example.com info sshd(pam_audit)[2147]: 01070417:6: AUDIT - user root - RAW: sshd(pam_audit): user=root(root) partition=[All] level=Administrator tty=ssh host=10.14.13.20 attempts=1 start="Mon Aug 5 17:02:30 2019" end="Mon Aug 5 17:02:54 2019".
-- ltm:Aug 5 17:02:42 bigip1.example.com warning tmsh[2200]: 01420013:4: Per-invocation log rate exceeded; throttling.
Conditions:
BIG-IP hostname contains periods or an FQDN:
[root@bigip1:Active:Standalone] log # tmsh list sys global-settings hostname
sys global-settings {
hostname bigip1.example.com
Impact:
Hostname is logged inconsistently. Some logs write the full hostname (FQDN), while other log files write only the hostname portion. This can make searching on hostname more complicated.
Workaround:
None.
Behavior Change:
Syslog-ng was using truncated hostname (without FQDN) while logging. This release adds fqdn use_fqdn(yes) in the syslog-ng template, so the system now logs the full hostname (FQDN).
Component: TMOS
Symptoms:
An agetty error message is output to the /var/log/secure log fil every 10 seconds while the instance remains on:
agetty[<process_id>]: /dev/tty0 ttyS0: No such file or directory.
Conditions:
This agetty error message is an issue on all BIG-IP Virtual Edition and Cloud instances. It is not configuration-dependent.
Impact:
This may fill the /var/log/secure log file. When /var/log is full, certain system services may degrade or become unresponsive (e.g., DNS).
Workaround:
Manually extend the /var/log logical volume.
For more information, see Increase disk space for BIG-IP VE :: https://clouddocs.f5.com/cloud/public/v1/shared/disk_space.html.
Component: TMOS
Symptoms:
If 'tmsh save sys config' is run simultaneously with 'tmsh save sys ucs <file>', there is the possibility of a race condition where a file gets scheduled to be added to the UCS file, but gets deleted by the save-config before it actually gets saved.
Conditions:
-- Run 'save sys config' at the same time as 'save sys ucs <file>' in tmsh.
-- Files are getting added by one tmsh command, yet deleted by the other. For example, when deleting a file that has not been saved to the configuration, while the system tried to create a UCS that contains that to-be-deleted file.
Note: There are many operations in which 'save sys config' is performed internally, so running the 'save sys ucs <file>' operation might encounter the timing error any time, even when you are not manually running 'save sys config'.
Impact:
The UCS is not created, and system posts messages similar to the following:
-- config/bigip_base.conf/: Cannot stat: No such file or directory.
-- Exiting with failure status due to previous errors.
-- Operation aborted.
This is a rare, timing-related occurrence. Even though the 'save sys ucs <file>' aborts and logs errors, simply re-running the command is likely to succeed.
Workaround:
Re-run the 'save sys ucs <file>' after it aborts. Nothing else needs to be changed or restored.
Component: TMOS
Symptoms:
V16.0.0 and v15.1.2 have a new SNMP MIB, sysVlanIfcStat, for VLAN statistics. Previously there were no ways to retrieve PVA statistics for each VLAN.
Conditions:
Viewing VLAN statistics through SNMP queries.
Impact:
PVA stats are not displayed.
Component: TMOS
Symptoms:
The tmsh command 'show net vlan' only displays statistics for each VLAN's interface, and not each VLAN's statistics.
Conditions:
This is encountered when running the command 'tmsh show net vlan'.
Impact:
You are unable to view PVA statistics.
Behavior Change:
Starting in v16.0.0, the command 'tmsh show net vlan' now outputs each vlan's stats retreived from TMM, including PVA stats, immediately following 'customer-tag' and right before the list of interfaces.
Example:
[root@localhost:Active:Standalone] config # tmsh sho net vlan
-------------------------------------
Net::Vlan: external
-------------------------------------
Interface Name external
Mac Address (True) f4:15:63:52:10:84
MTU 1500
Tag 4094
Customer-Tag
| Incoming Discard Packets 2
| Incoming Error Packets 0
| Incoming Unknown Proto Packets 0
| Outgoing Discard Packets 0
| Outgoing Error Packets 0
| HC Incoming Octets 28.8M
| HC Incoming Unicast Packets 121.2K
| HC Incoming Multicast Packets 121.2K
| HC Incoming Broadcast Packets 0
| HC Outgoing Octets 14.4M
| HC Outgoing Unicast Packets 8
| HC Outgoing Multicast Packets 121.2K
| HC Outgoing Broadcast Packets 45
| PVA Incoming Packets 5
| PVA Incoming Octets 443
| PVA Outgoing Packets 7
| PVA Outgoing Octets 3.2K
-----------------------
| Net::Vlan-Member: 1.1
-----------------------
| Tagged no
| Tag-Mode none
--------------------------------------------------------------------
| Net::Interface
| Name Status Bits Bits Pkts Pkts Drops Errs Media
| In Out In Out
--------------------------------------------------------------------
| 1.1 up 133.5M 31.5K 129.3K 17 36 0 10000SR-FD
-------------------------------------
Net::Vlan: internal
-------------------------------------
Interface Name internal
Mac Address (True) f4:15:63:52:10:85
MTU 1500
Tag 4093
Customer-Tag
| Incoming Discard Packets 0
| Incoming Error Packets 0
| Incoming Unknown Proto Packets 0
| Outgoing Discard Packets 0
| Outgoing Error Packets 0
| HC Incoming Octets 28.8M
| HC Incoming Unicast Packets 121.2K
| HC Incoming Multicast Packets 121.2K
| HC Incoming Broadcast Packets 1
| HC Outgoing Octets 14.4M
| HC Outgoing Unicast Packets 15
| HC Outgoing Multicast Packets 121.2K
| HC Outgoing Broadcast Packets 17
| PVA Incoming Packets 7
| PVA Incoming Octets 3.2K
| PVA Outgoing Packets 5
| PVA Outgoing Octets 443
-----------------------
| Net::Vlan-Member: 1.2
-----------------------
| Tagged no
| Tag-Mode none
--------------------------------------------------------------------
| Net::Interface
| Name Status Bits Bits Pkts Pkts Drops Errs Media
| In Out In Out
--------------------------------------------------------------------
| 1.2 up 133.5M 14.0K 129.3K 22 16 0 10000SR-FD
This feature is also available as of v15.1.2 on an 'opt-in' basis; default output is unchanged. A new db variable is available on v15.1.2 for controlling the output format. To change the output of 'tmsh show net vlan' so that it provides the new stats, run the command:
tmsh modify sys db vlan.pva.stats value enable
Output can be changed back to the old format at any time by running:
tmsh modify sys db vlan.pva.stats value disable
Component: Protocol Inspection
Symptoms:
Under certain conditions, TMM may crash while processing traffic with a FastL4 virtual with the Protocol Inspection Profile attached.
Conditions:
-FastL4 Virtual
-Protocol Inspection Profile
Impact:
TMM crash, leading to a failover event.
Workaround:
None.
Component: TMOS
Symptoms:
Issuing the command 'tmsh save sys config' results in a question when display threshold is set and when management routes are configured. There is no prompt when no management routes are configured. This question is posted only when management-routes are configured, and does not appear when other provisioning commands are issued and the config is saved.
Conditions:
1. Set the cli preference display-threshold to a smaller value than the default.
2. Create management routes.
3. Issue the following command:
tmsh save sys config
Impact:
When there are more items configured than the threshold, the system presents a question:
Display all <number> items? (y/n)
Scripts are stopped until the prompt is answered.
Workaround:
To prevent the question from popping up, set display threshold to 0 (zero).
In the case of this script, you can also delete the management route definitions to prevent the question from being asked.
Component: Access Policy Manager
Symptoms:
Requests that evaluate to the same subsession can be processed concurrently in some cases
Conditions:
-- Per-Request policy with subroutines.
-- Duplicate requests are sent that match existing subsession gating criteria.
Impact:
The request gets aborted with error messages in /var/log/apm:
apmd_plugin.cpp func: "serialize_apmd_reply()" line: 495 Msg: AccessV2 agent execution error 4.
Workaround:
None.
Component: Global Traffic Manager (DNS)
Symptoms:
On rare occasions, some resources are marked 'unavailable', with a reason of 'big3d: timed out' because gtmd fails to process some probe responses sent by the instance of big3d that is running on the same BIG-IP system.
Conditions:
The monitor response from big3d sent to the gtmd on the same device is being lost. Monitor responses sent to other gtmds are sent without issue. The conditions under which this occurs have not been identified.
Impact:
Some resources are marked 'unavailable' on the affected BIG-IP system, while the other BIG-IP systems in the sync group mark the resource as 'available'.
Workaround:
Restart gtmd on the affected BIG-IP system.
Component: TMOS
Symptoms:
The output file from tcpdump generation is named support.tcpdump even though it is a compressed file.
Conditions:
Whenever tcpdump is generated and downloaded.
Impact:
You must rename the file with the correct file extension and then decompress it to access the .dmp files.
Workaround:
Rename the downloaded file from support.tcpdump to <filename>.tar.gz and decompress it.
Behavior Change:
The tcpdump file has a different name and file extension - support.tcpdump.tar.gz
Component: Access Policy Manager
Symptoms:
tmm cores. System posts a message:
access::session exists is used during CLIENT_CLOSED iRule event.
Conditions:
-- Client has closed the connection.
-- iRule function tries to check on a closed session.
-- An 'access session::exists' command is used inside the iRule event CLIENT_CLOSED.
Impact:
tmm may core. Traffic disrupted while tmm restarts.
Workaround:
Do not use the iRule command 'access session::exists' inside CLIENT_CLOSED.
Component: TMOS
Symptoms:
Configuring EngineID for SNMPv3 does not take effect until
the SNMP and Alert daemons are restarted.
Conditions:
Configure the EngineID for SNMPv3 using the tmsh command:
modify sys snmp include 'EngineType n'
Impact:
The SNMPv3 value does not take effect.
Workaround:
Restart the daemons after changing the EngineID:
restart /sys service snmpd
restart /sys service alertd
Note: The SNMP daemon should be restarted before the Alert daemon.
Component: TMOS
Symptoms:
Under certain circumstances, the Traffic Management Shell (tmsh) can consume more system memory than expected.
Conditions:
Multiple users or remote processes connecting to the BIG-IP administrative command-line interface.
Impact:
Increased control plane pressure. Various delays may occur in both command-line and GUI response. Extreme instances may cause one or more processes to terminate, with potential disruptive effect. Risk of impact from this issue is increased when a large number of automated tmsh sessions are created.
Workaround:
For users with administrative privilege (who are permitted to use the 'bash' shell), the login shell can be changed to avoid invoking tmsh when it may not be needed:
tmsh modify /auth user ADMINUSERNAME shell bash
Component: Application Security Manager
Symptoms:
You cannot create non-ASCII entities (such as URLs and parameters) in a non-UTF-8 policy using REST.
Conditions:
-- The policy is configured for an encoding other than UTF-8.
-- Attempting to create non-ASCII entries using REST.
Impact:
You cannot create an entity (such as a URL or parameter) which contains non-ASCII characters using REST.
Workaround:
Use UTF-8.
Component: TMOS
Symptoms:
Syslog-ng consumes more than 95% CPU starving other processes of CPU time. This leads to eventual mcpd crash with core.
Conditions:
Configuring nonexistent local IP addresses and remote log server.
Impact:
Abnormal CPU usage. Potential eventual mcpd crash with core.
Workaround:
To mitigate the issue, you can use either of the following:
-- Follow these two steps:
1. Remove the remote log server from the configuration.
2. Replace the nonexistent local IP addresses with self IP addresses.
-- Configure the remote destination host with a unique parameter in the configuration so that syslog does not get confused if there are multiple entries:
udp(192.0.2.1 port(512) localip(192.0.2.200) persist-name(r1));
udp(192.0.2.1 port(512) localip(192.0.2.201) persist-name(r2));
udp(192.0.2.100 port(512) localip(192.0.2.200) persist-name(r3));
udp(192.0.2.100 port(512) localip(192.0.2.201) persist-name(r4));
Component: Application Security Manager
Symptoms:
When using AWAF or MSP license, you cannot use the BIG-IP Configuration Utility to set rule to control ASM in an LTM policy.
Conditions:
- AWAF or MSP license
Impact:
Admin cannot use the BIG-IP Configuration Utility create LTM policy that controls ASM, and must use TMSH.
Workaround:
Use TMSH to create the rule instead of GUI:
For example:
create ltm policy Drafts/test99 controls add { asm } requires add { http } rules add { rule1 { actions add { 0 { asm enable policy dummy2 }} ordinal 1 }}
Component: Access Policy Manager
Symptoms:
Installation of EPSEC package via tmsh fails with error 'Configuration error: Invalid mcpd context, folder not found (/Common/EPSEC/Images)"
Conditions:
-- EPSEC package was never installed on the BIG-IP machine.
-- Running 'tmsh create apm epsec epsec-package <package_name>.iso local-path /shared/apm/images/<package_name>.iso'
Impact:
First time installation of EPSEC package through tmsh fails.
Workaround:
You can do a first-time installation of EPSEC with the following commands:
tmsh create sys folder /Common/EPSEC
tmsh create sys folder /Common/EPSEC/Images
tmsh install Upload/<package_name>.iso
Component: TMOS
Symptoms:
After FIPS 140-2 license is installed on BIG-IP FIPS-certified hardware devices, the system halts while booting upon performing the FIPS integrity check.
Console shows messages similar to:
Starting System Logger Daemon...
[ OK ] Started System Logger Daemon.
[ 14.943495] System halted.
Conditions:
-- The BIG-IP device has a license that includes the FIPS 140-2 option (FIPS full-box license).
-- System element monitored by FIPS 140-2 integrity check has changed.
-- The device is rebooted.
Impact:
The device halts and cannot be used.
Workaround:
Workaround:
[1] Connect a terminal to the BIG-IP serial console port.
[2] From the console, enter the GRUB menu and boot into a partition that does not have a FIPS 140-2-enabled license, or into TMOS Maintenance.
[3] Mount config from the inactive partition (see K51222154: Mounting the filesystem of an inactive partition :: https://support.f5.com/csp/article/K51222154) that was halted, and examine the contents of /config/f5_public/fipserr, which shows the files that were changed, leading to failure of the FIPS 140-2 license-enabled partition.
[4] Restore those files to their original ones.
[5] Truncate the inactive partition's /config/f5_public/fipserr, e.g., by running:
cat /dev/null > /mnt/test/f5_public/fipserr
[6] Reboot.
If the system still halts, repeat from Step [1] above, until this no longer happens.
Component: TMOS
Symptoms:
If the configsync IP address of the device is configured to be an IPv6 address, changes in ASM configuration do not synchronize across the cluster.
Conditions:
Devices in a Device Group have an IPv6 address set as their configsync IP address.
Impact:
ASM configuration does not synchronize across the Device Group.
Workaround:
Set the configsync IP address to be an IPv4 address and restart the asm_config_server process. To restart the asm_config_server process, run the following command:
pkill -f asm_config_server
Component: Local Traffic Manager
Symptoms:
TMM crash when running show ltm clientssl-proxy cached-certs virtual [name] clientssl-profile [name]
Conditions:
-- Running the command:
show ltm clientssl-proxy cached-certs virtual [name] clientssl-profile [name].
- The command is terminated by the client connection, aborting with CTRL-C.
Impact:
Traffic disrupted while tmm restarts.
Workaround:
Do not terminate tmsh show commands with CTRL-C.
Component: Local Traffic Manager
Symptoms:
After a Full Configuration Sync is performed in a device cluster, Ephemeral (FQDN) nodes on peers to the device initiating the Configuration Sync will report their status as Unknown with monitor status of Unchecked.
Note: The nodes are still monitored properly by the peer devices even though they are not reported as such.
Conditions:
-- Full configuration sync performed in a device cluster.
-- Ephemeral (FQDN) nodes configured.
Impact:
Monitor status on the peer devices is reported incorrectly.
Workaround:
Any of the following three options will correct reporting status on the peer devices:
-- Restart bigd
-- Cause monitoring to the FQDN nodes to fail for at least one probing interval, and then restore monitoring accessibility.
-- Disable and then re-enable the FQDN node
Each of these workarounds results in the reported status of the FQDN node on the peer reporting correctly again. The workarounds do not prevent a subsequent configuration sync from placing the FQDN nodes back into Unknown status on peers, however.
Component: Application Security Manager
Symptoms:
The system does not always show violations in request log when post request is large.
Conditions:
A large post request with many parameters is sent.
Impact:
Although the violations is handled correctly, it is not reported.
Workaround:
Disable learning mode.
The internal parameter pb_sampling_high_cpu_load can define what is seen as high CPU load above which sampling does not take place. The default is 60.
-- Using a lower value reduces the chances of sampling data.
-- Using 0 makes sampling never happen and thus this issue does not occur (this slows down automatic policy building).
Component: TMOS
Symptoms:
If multiple FQDN nodes and corresponding pool members are created, with FQDN names that resolve to the same (or a common) IP address, you may not be able to delete any of the affected FQDN nodes even after its corresponding FQDN pool member has been deleted.
Conditions:
This occurs under the following conditions
-- Multiple FQDN template nodes exist with FQDN names that resolve to the same (or a common) IP address.
-- FQDN pool members exist for each FQDN template node, with corresponding ephemeral pool members for each which share the same IP address.
-- One of the FQDN pool members is removed from its pool.
-- You attempt to delete the corresponding FQDN template node.
Impact:
The FQDN template node remains in the configuration and cannot be deleted, while an ephemeral node or pool member exists with an IP address corresponding to that FQDN name.
Workaround:
To work around this issue:
1. Remove all remaining conflicting FQDN pool members (with FQDN names that resolve to the shared/conflicting IP address).
2. Delete the desired FQDN node.
3. Re-create the remaining FQDN pool members to replace those removed in step 1.
Component: Local Traffic Manager
Symptoms:
The radius monitor marks a pool member down if it does not respond within 6 seconds, regardless of the interval or timeout settings in the monitor configuration.
Conditions:
A radius monitor is used, and the pool member takes more than 6 seconds to respond to a radius request.
Impact:
The pool member may be marked down incorrectly if the monitor interval is configured to be greater than 6 seconds.
Workaround:
There is no workaround at this time.
Component: TMOS
Symptoms:
If an unpopulated front panel interface is disabled, then an SFP inserted and the interface re-enabled, TMSH will continue to display the interface as 'disabled' in 'tmsh show net interface output' commands.
Conditions:
-- BIG-IP appliance or blade.
-- Unpopulated front panel interface is disabled.
-- SFP inserted and the interface re-enabled.
-- Running the command: tmsh show net interface output.
Impact:
Output of the command shows the interface is disabled even though it is enabled and fully operational.
Workaround:
This issue is cosmetic; the interface is functional so it may be used.
To correctly identify the enabled/disabled state of the interface, use the following command: tmsh list net interface
Component: TMOS
Symptoms:
There is no limit to the number of users that can login concurrently onto a BIG-IP system.
Conditions:
Multiple users are logged into the BIG-IP device through SSH at the same time.
Impact:
System can potentially run out of memory.
Workaround:
Provide a way to limit the number of concurrent user SSH sessions.
Behavior Change:
There are new db variables available for specifying SSH session limits, overall, per-user, and for a specific user.
-- Command: modify sys global-settings ssh-session-limit [enable/disable]
Specifies enable/disable of ssh session limit feature.
+ Enables the feature; feature is functional with default values.
+ Defaults: feature is not enabled for admin/root privileged user.
+ Total session limit for all users is 10 sessions.
-- Command: modify sys global-settings ssh-root-session-limit [enable/disable]
Specifies enable/disable of SSH session limit feature for root user.
+ Enables feature for admin/root privileged user.
+ Total session limit for all users is still 10 sessions.
-- Command: modify sys global-settings ssh-max-session-limit <value>
Specifies a global maximum number of SSH sessions.
+ Changes the default global setting limit of 10 to the specified value.
-- Command: modify sys global-settings ssh-max-session-limit-per-user <value>
Specifies a global maximum number of SSH sessions for each user.
+ Sets the maximum session limit per user.
+ Total sessions on the system are still enforced by the setting for ssh-max-session-limit.
-- Command: create auth user <> session-limit <value>
Specifies a user-specific SSH sessions limit.
+ Sets the maximum number of sessions for a particular user.
+ Total sessions on the system are still enforced by the setting for ssh-max-session-limit.
Component: Local Traffic Manager
Symptoms:
iRulesLX does not follow current best practices and should be updated to ensure layered protections.
Conditions:
-iRulesLX in use
Impact:
iRulesLX does not follow current best practices.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
On a standby BIG-IP system, an ephemeral pool-members state remains user-down after re-enabling an FQDN node on the primary system.
Conditions:
Re-enabling a forced-down FQDN node on the primary system.
Impact:
On the standby system, the ephemeral pool-members are in state: user-down, (forced-down in GUI).
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
bigd restarts when loading a UCS for the first time, where the load succeeds; and no related messages are reported in /var/log/ltm; and no bigd core file is produced.
Conditions:
bigd loads a UCS file for the first time, such as after the command:
tmsh load sys ucs no-license keep-current-management-ip no-platform-check
Impact:
The UCS file is correctly reloaded, and bigd restarts with the loaded configuration. No bigd core is produced, and no related messages are found in /var/log/ltm. After restart, bigd performs all system functions as expected.
Workaround:
System runs as expected after the bigd restart, and the user need not take any action.
Component: Local Traffic Manager
Symptoms:
If the BIG-IP system has a web-acceleration which provides a number of caching and optimization options suitable for HTTP/1.1. It uses 'Connection: Keep-Alive' header on a server side, which results in appearance of 'Keep-Alive' header in a response. Such a HTTP header was adopted by the industry but not standardized. When a web-acceleration profile is configured and provides a response, Safari clients do not accept responses with a such header and reject those with a RST_STREAM message.
Conditions:
-- BIG-IP has a virtual server with HTTP/2 profile and a web-acceleration profile.
-- A pool member responds with 'Keep-Alive' header in the following format: Keep-Alive: timeout=<number>, max=<number>.
Impact:
A response to a request is rejected, which might cause incorrect rendering of HTTP page.
Workaround:
Use an iRule to remove the Keep-Alive header:
when HTTP_RESPONSE_RELEASE {
HTTP::header remove keep-alive
Alternatively use an LTM Policy where this header is removed from a server's response.
Component: TMOS
Symptoms:
-- UCS archive restore fails
-- The Traffic Management Shell (TMSH) and/or /var/log/ltm file show following error message:
01071769:3: Decryption of the field (privatekey) for object (9717) failed. Unexpected Error: Loading configuration process failed.
Conditions:
- Restoring configuration from UCS.
- The UCS is being restored on a different BIG-IP system with a different master key.
Impact:
-- The UCS configuration is not applied.
-- The BIG-IP is not in a fully operational state.
Workaround:
If you encounter this error and dynad is not in use (dynamic debug) you can manually edit bigip_base.conf.
1. Locate the dynad config in /config/bigip_base.conf file:
For example, the dynad config will look like:
sys dynad key {
key $M$jV$VX7HMp5q346nsTYDYFPnYdJLrBPyQSCrDTJYAz4je7KXJAC38fxtDJL35KtF66bq
2. Modify the dynad configuration lines to:
sys dynad key {
key "test"
3, Save the updated bigip_base.conf file
4. Load the configuration with command: tmsh load sys config
Component: TMOS
Symptoms:
Warning similar to below, referencing a non-floating self IP:
Virtual address (/Common/10.10.10.10) shares address with floating self IP (/Common/10.10.10.10), so traffic-group is being kept at (/Common/traffic-group-local-only)
Conditions:
Virtual Server is defined using the same IP address as a non-floating self IP.
Impact:
Virtual Server does not fail over with floating traffic group as expected.
Component: Access Policy Manager
Symptoms:
In configurations with large numbers of virtual servers or other entities, the apmd, oauth, and localdbmgr processes may consume large amounts of system resources.
Conditions:
-- Large configuration.
-- APM provisioned.
-- Multiple traffic groups exacerbate the effect.
Impact:
Heavy use of odd-numbered CPU cores may slow all control-plane operations, including user-interface response.
Workaround:
None known.
Component: TMOS
Symptoms:
TACACS Shared Key is not encrypted in the DB key and is visible to admin and a read-only user.
Conditions:
Configure TACACS+ auditing forwarder.
Impact:
Exposes sensitive information.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
The dynconfd daemon may crash with a core and restart when processing a DNS query when there are multiple DNS name servers configured, or when the list of DNS name servers is changed.
Conditions:
This may occur rarely when FQDN nodes are configured and multiple DNS name servers are configured, including when a name server is added to or removed from the system DNS configuration while a DNS query is active.
Impact:
Resolution of FQDN names for FQDN nodes and pool members may be briefly interrupted while the dynconfd daemon restarts. This may cause a delay in propagation of DNS zone changes to the BIG-IP configuration.
Workaround:
This issue occurs rarely. There is currently no known workaround.
Component: Local Traffic Manager
Symptoms:
DHCP relay agent does not have server flows connecting to all active DHCP servers after a while.
Conditions:
- More than one DHCP servers configured for a DHCP virtual.
- Server flows timeout in 60 seconds
Impact:
DHCP server traffic not load balanced.
Workaround:
None.
Component: Service Provider
Symptoms:
Command returns "No route to host" error.
Conditions:
Running diadb, sipdb, genericmsgdb or lsndb when only a single TMM is running.
Impact:
Unable to query SIP, Diameter, Generic-Message and LSN information from its corresponded DB query tool.
Workaround:
Tools for querying various DBs now work regardless of the number of TMMs.
Component: Service Provider
Symptoms:
When adding "GTP::ie insert" and "GTP::ie append" without "-message" option to iRule, there is warning message:
[The following errors were not caught before. Please correct the script in order to avoid future disruption. "unexpected end of arguments;expected argument spec:VALUE"1290 38]
Conditions:
Using "GTP::ie insert" or "GTP::ie append" command without "-message" option
Impact:
The commands still be executed during runtime but the warning message may confuse user.
Component: Global Traffic Manager (DNS)
Symptoms:
Under certain conditions a virtual server may change status to yellow despite proper function of pool members.
Conditions:
-- LTM virtual server
-- Connection Rate Limit Mode enabled
Impact:
Virtual server stops responding.
Workaround:
None.
Component: Service Provider
Symptoms:
The following commands in iRules are created and available but not officially tested and approved:
GTP::ie set instance/value
GTP::ie insert
GTP::ie append
GTP::ie remove
Conditions:
Using the following iRule commands:
GTP::ie set instance/value
GTP::ie insert
GTP::ie append
GTP::ie remove
Impact:
Although you can use these iRule commands, their functionality has not been tested and approved.
Workaround:
None.
Behavior Change:
Certified pre-existing iRules:
-- GTP::ie set instance <ie-path> <instance>
Assigns <instance> to the information element (IE) instance at <ie-path>.
-- GTP::ie set value <ie-path> <value>
Assigns <value> to the IE value at <ie-path>.
-- GTP::ie insert <ie-path> <type> <instance> <value>
Inserts a new IE of type <type> and instance <instance> with value <value> at <ie-path>
-- GTP::ie append [<ie-path>] <type> <instance> <value>
Appends a new IE of type <type> and instance <instance> with value <value> to the end of embeded IE of grouped-IE specified by <ie-path> or to the end of message if the grouped-IE <ie-path> is absent.
-- GTP::ie remove <ie-path>
Removes IE specified by <ie-path>.
Component: Application Security Manager
Symptoms:
While 'Query String' contains masked sensitive parameter value the 'Referer' header sensitive parameter value is exposed.
Conditions:
-- Sensitive parameter is defined in: 'Security :: Application Security : Parameters : Sensitive Parameters'.
-- 'Query String' contains the defined sensitive parameter.
Impact:
"Referer" header contains unmasked value of the sensitive parameter.
Workaround:
Enable 'Mask Value in Logs' in: 'Security :: Application Security : Headers : HTTP Headers :: referer'.
Component: Access Policy Manager
Symptoms:
APM resets connections and reports an ERR_ARG from a simple web request.
Conditions:
-- APM profile with User Identification Method as HTTP.
-- APM profile with User Identification Method as OauthToken.
-- HTTP traffic arrives with certain types of Authorization headers.
Impact:
Connections are reset and APM logs ERR_ARG, which is not helpful for understanding the cause.
Workaround:
iRule workaround:
when HTTP_REQUEST {
if { [HTTP::header "Authorization"] contains "Bearer" && [string tolower [HTTP::header "User-Agent"]] contains "onenote" } {
HTTP::header replace Authorization [string map {"Bearer" ""} [HTTP::header Authorization]]
APM no longer resets connections and reports an ERR_ARG from a simple web request.
Component: Global Traffic Manager (DNS)
Symptoms:
Query of "MX" "any" RR of CNAME wide IP results in NXDOMAIN.
Conditions:
A CNAME wide IP and a dnx with parent zone.
For example, CNAME wide IP for www.siterequest.com and a dnx zone for siterequest.com.
Impact:
Cache resolvers will remember NXDOMAIN for the entire name. So clients talking to those caches asking for A/AAAA records may actually get NXDOMAIN responses until the negative cache expires.
Workaround:
Option 1: Create a related "www.siterequest.com" txt record in ZoneRunner
Option 2: Create a ltm virtual server iRule, similar to this:
when DNS_RESPONSE {
if { [DNS::question name] eq "www.siterequest.com" } {
if { [DNS::header rcode] eq "NXDOMAIN" } {
DNS::header rcode NOERROR
DNS::authority clear
return
Component: Application Security Manager
Symptoms:
Users report their access is blocked; when you look at the error log, you see CSRF errors.
Conditions:
-- CSRF enabled on ASM.
-- ASM client is using a mobile device.
Impact:
Client is blocked.
Workaround:
None.
Component: TMOS
Symptoms:
Configuration changes in quick succession might generate sync change messages faster than the receiving BIG-IP system can parse them. The sending BIG-IP system's queue for its peer connection fills up, mcp fails to allocate memory, and then the system generates a core file.
Conditions:
Configuration changes in quick succession that might generate sync-change messages.
Impact:
Core file and sync operation does not complete as expected. The possibility for this occurring depends on the size and complexity of the configuration, which impacts the time required to sync, and the traffic load occurring at the time of the sync operation.
Workaround:
None.
Component: TMOS
Symptoms:
Devices do not have matching configuration, but system reports device group as being 'In Sync'.
Conditions:
This occurs when the following conditions are met:
-- Device Service Cluster Device Group with incremental sync is enabled.
-- A ConfigSync operation occurs where a configuration transaction fails validation.
-- A subsequent (or the final) configuration transaction is successful.
Impact:
The BIG-IP system incorrectly reports that the configuration is in-sync, despite the fact that it is not in sync. You might experience various, unexpected failures or unexplained behavior or traffic impact from this.
Workaround:
Turn off incremental sync (by enabling 'Full Sync' / 'full load on sync') for affected device groups.
Once the systems are in sync, you can turn back on incremental sync, and it will work as expected.
Component: TMOS
Symptoms:
iControl produces an HTTP 404 - Not Found error message while creating the BIG-IP DNS topology record successfully.
Conditions:
Creating GTM topology record without using full path via iControl.
Impact:
Resulting code/information is not compatible with actual result.
For a post request, the create command and the list command are formed and executed, and the name in the curl request and the name in the list response are compared to verify whether or not it is the actual object. When a create command is executed with properties that are not fullPath (e.g., in iControl), it still creates the object with fullPath. So list returns the name with fullPath and compares it with the name that does not contain the fullPath, and the comparison fails because the names do not match.
Workaround:
Use the full path when creating BIG-IP DNS topology records using iControl.
Component: Access Policy Manager
Symptoms:
Reboot a 4-blade vCMP guest. Now, only the master key for catalog remained. All subkeys are missing.
Conditions:
This can occur intermittently during a reboot in a multi-blade vCMP guest configured with APM.
Impact:
Some Access subkeys may be missing after the reboot.
Workaround:
Reboot the primary blade.
Component: TMOS
Symptoms:
Sync fails if you create/import a file object and delete it before triggering manual sync; ltm logs contain messages similar to the following:
Standby:
-- err mcpd[7339]: 01070712:3: Caught configuration exception (0), Failed to sync files..
-- err mcpd[7339]: 01071488:3: Remote transaction for device group /Common/test to commit id 42 6079477704784246664 /Common/test failed with error 01070712:3: Caught configuration exception (0), Failed to sync files...
Active:
-- err mcpd[6319]: 0107134a:3: File object by name (/Common/filename) is missing.
Conditions:
This occurs when the following conditions are met:
-- BIG-IP systems configured for high availability (HA) are not configured to sync automatically, and incremental synchronization is enabled (these are the default settings).
-- One or more file objects are created and deleted before performing a sync from Active to Standby.
Impact:
Sync fails.
Workaround:
When you create/add a file object, make sure to sync before deleting it.
If a system is already in this state, perform a full sync and overwrite the configuration, as described in K13887: Forcing a BIG-IP device group member to initiate a ConfigSync operation :: https://support.f5.com/csp/#/article/K13887.
Component: TMOS
Symptoms:
The imish (vtysh) utility dumps core while checking help strings for the OSPF6 vertex-threshold command from OSPF mode.
Conditions:
-- Routing enabled.
-- Configuring OSPF vertex-threshold for either IPv4 or IPv6.
-- Requesting help for the OSPF6 vertex-threshold command.
Impact:
There is an imish (vtysh) crash.
Workaround:
None.
Known Issues in BIG-IP v15.1.x
Component: TMOS
Symptoms:
Mcpd might core and restart if it fails to process a query for all node statistics in less than 5 minutes.
There is more then one avenue where node statistics would be queried.
The BIG-IP Dashboard for LTM from the GUI is one example.
Conditions:
Thousands of FQDN nodes and pools with FQDN pool members and a query for all node statistics.
Impact:
Mcpd restarted which will cause services to failover. Traffic and configuration disrupted while mcpd restarts.
Component: Access Policy Manager
Symptoms:
The General Customization interface for Endpoint Security in the GUI cannot be used for Access Profile with modern customization due to interface error.
Conditions:
-- Access Profile with modern customization
-- General Customization interface for Endpoint Security
Impact:
You are unable to modify Endpoint Security interface strings
Component: Application Security Manager
Symptoms:
ASM clients encounter poor network performance.
Conditions:
A replacement of data with * happens due to data guard.
Impact:
Slow responses.
Workaround:
Disable data guard or block the data instead of masking it.
Component: Application Security Manager
Symptoms:
Clients encounter a longer time to respond from the BIG-IP
Conditions:
-- One of the following features is enabled:
- convictions
- csrf
- ajax.
-- The response is HTML
-- The response has many tags
Impact:
Slow performance. May lead to a bd crash on specific responses. Traffic disrupted while bd restarts.
Component: Advanced Firewall Manager
Symptoms:
When a client sends a DNS request to a NON EDNS capable server, the server may send a legitimate response with RCODE FORMERR and no DNS data. The MALFORM DNS vector blocks those requests.
Conditions:
-- The client sends a DNS request to NON EDNS capable server
-- The server replies with RCODE FORMERR and no DNS data.
Impact:
AFM erroneously detects an attack and mitigates it, and the client does not get a responses from the EDNS server
Workaround:
Disable DNS MALFORM vector mitigation or put the EDNS server in allow list.
Component: TMOS
Symptoms:
A virtual server configuration is changed unexpectedly
Conditions:
- create virtual server with two client-ssl profiles
- modify same virtual server in Protected Objects panel
Impact:
Virtual servers client-ssl profiles getting removed if you have more then one profile
Component: TMOS
Symptoms:
Upon booting a BIG-IP hardware system running an Engineering Hotfix version of BIG-IP v14.1.0 or later, messages of the following form may be logged in the LTM log file (/var/log/ltm):
err tpm-status[####]: System Integrity Status: Invalid
info tpm-status-check[####]: System Integrity Status: Invalid
In addition, a message similar to the following may appear on the serial console while the system is booting:
[ ###.######] tpm-status-check[####]: Checking System Integrity Status
[ ###.######] tpm-status-check[####]: sh: /bin/rpm: Permission denied
[ ###.######] tpm-status-check[####]: tpm-status-check: System Integrity Status: Invalid
Similar messages appear when viewing the status of the tpm-status-check service via the systemctl utility:
# systemctl -l status tpm-status-check.service
* tpm-status-check.service - F5 Trusted Platform Module
Loaded: loaded (/usr/lib/systemd/system/tpm-status-check.service; static; vendor preset: enabled)
Active: failed (Result: exit-code) since <...>
Main PID: #### (code=exited, status=1/FAILURE)
<...> tpm-status-check[####]: Checking System Integrity Status
<...> tpm-status-check[####]: sh: /bin/rpm: Permission denied
<...> tpm-status[####]: TPM Status Version 15.1.1.0.6.6
<...> tpm-status[####]: TMOS BIG-IP 15.1.1-0.0.6.0
<...> tpm-status[####]: BIOS 0614 v3.10.032.0
<...> tpm-status[####]: BIOS SIRR 2019-05-30_08-46-02
<...> tpm-status-check[####]: tpm-status-check: System Integrity Status: Invalid
<...> systemd[1]: tpm-status-check.service: main process exited, code=exited, status=1/FAILURE
<...> systemd[1]: Unit tpm-status-check.service entered failed state.
<...> systemd[1]: tpm-status-check.service failed.
However, checking the System Integrity Status using the "tpm-status" or "tmsh run sys integrity status-check" command shows "System Integrity Status: Valid"
Conditions:
This may occur:
-- running affected BIG-IP v14.1.0 or later releases with an Engineering Hotfix containing fixes for ID893885 and ID946745
-- on hardware platforms that include a Trusted Platform Module (TPM), including:
-- BIG-IP i2000, i4000, i5000, i7000, i10000, i11000, i15000 Series appliances
-- VIPRION B4450 blades
Impact:
The tpm-status-check service inaccurately indicates that the System Integrity Status is not Valid.
This is incorrect, and conflicts with the accurate System Integrity Status provided by the "tpm-status" utility and "tmsh run sys integrity status-check" command.
Workaround:
Use the "tpm-status" utility or "tmsh run sys integrity status-check" command to observe the correct System Integrity Status.
Component: Local Traffic Manager
Symptoms:
The number of native open connections in the SSL profile shows a value that is higher than expected.
Conditions:
SSL renegotiation is enabled. Other conditions are unknown.
Impact:
The SSL stats are incorrectly reading higher than expected.
Workaround:
Disable SSL renegotiation.
Component: TMOS
Symptoms:
When installing an ASU file, if there is not enough disk space in /var, the clntcp update file might become corrupted, causing datasyncd to be offline (and thus cause the entire BIG-IP offline)
Conditions:
-- ASM/FPS provisioned.
-- Installing ASU file.
-- Not enough space in /var partition.
Impact:
Device goes offline.
Workaround:
Delete the update file:
tmsh delete security datasync update-file /Common/datasync-global/update-file-clntcap_update_ (auto complete)
Component: Application Security Manager
Symptoms:
When using parent policies with learning enabled in an auto-sync device group, in some use cases deleting an ignored suggestion on a parent policy will cause a full sync to occur.
This can cause unexpected delays in configuration being synchronized between peers, and in the event of repeated instances in quick succession could fill the /var partition
Conditions:
1) Device Group with ASM and auto-sync enabled
2) Parent Policies with learning are in use.
3) Ignored Suggestions are deleted on the parent policy after they have 0 suggesting children left.
Impact:
ASM configuration requests a full sync which can cause unexpected slowness in config synchronization and may fill the /var partition if done multiple times in quick succession.
A full /var partition can lead to bd cores.
Component: Application Security Manager
Symptoms:
BD crash when checking ICAP job after ICAP response
Conditions:
BD is used with ICAP feature
Impact:
Traffic disrupted while BD restarts.
Workaround:
Component: Application Security Manager
Symptoms:
A request is not logged in the local and/or remote logs.
Conditions:
A request has evasions detected on very large parameters.
Impact:
A missing request in the log.
Workaround:
Component: Application Security Manager
Symptoms:
False positive parameters are being detected in the payload and enforced wrongly.
Conditions:
The URL is not defined (also not as wildcard - not defined at all) and the request has a payload.
Impact:
False positive enforcement - may lead to wrong violations and wrong blocking of requests.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
In case of a HEAD request, BIG-IP fails to set up a new connection to the server with the HTTP::retry iRule.
Conditions:
1.) Basic HTTP profile is configured on BIG-IP
2.) BIG-IP sends the HEAD request to the server and gets error response
3.) iRule with HTTP::retry is configured
4.) The system is using the default (non-debug) TMM version
Impact:
BIG-IP might send the retry HEAD request after the connection is closed, more specifically after the server has sent a FIN, the retry is leaked on the network.
Component: Application Security Manager
Symptoms:
BD crashes when handling JSON payloads that does not conform with JSON content profile
Conditions:
JSON content profile is used in BD
Impact:
Traffic disrupted while BD restarts.
Workaround:
Component: Local Traffic Manager
Symptoms:
Unable to retrieve link statistics via SNMP OID gtmLinkStatRate
config # snmpwalk -c public localhost F5-BIGIP-GLOBAL-MIB::gtmLinkStatRate
F5-BIGIP-GLOBAL-MIB::gtmLinkStatRate = No Such Object available on this agent at this OID
Conditions:
A BIG-IP DNS/LC system configured with link objects.
Try to do an snmpwalk for F5-BIGIP-GLOBAL-MIB::gtmLinkStatRate which is not successful.
Impact:
BIG-IP DNS system link statistics cannot be retrieved via SNMP.
Workaround:
No workaround.
Component: Application Security Manager
Symptoms:
Websocket frames receive a close frame
Conditions:
-- ASM provisioned
-- ASM policy attached to a virtual server
-- Websocket profile attached to a virtual server
-- ASM policy transparent mode enabled
Impact:
Websocket frame blocked in transparent mode
Workaround:
Change signatures blocking settings to Learn = Yes, Alarm=Yes, Block=No
Component: Local Traffic Manager
Symptoms:
Arp requests not resolved for snatpool members and traffic does not go through when the primary blade becomes offline.
Conditions:
-- VIPRION platforms serving as AAA and Diameter virtual server to load-balance.
-- route-domain configured other than 0.
-- Radius authentication pool and snatpool are configured.
-- Primary blade goes offline and new Primary is not elected.
Impact:
Traffic failure when primary became offline.
Workaround:
Disable primary blade which is offline.
Component: Global Traffic Manager (DNS)
Symptoms:
Configurations that use a 'DNS Cache' or 'Network DNS Resolver' are affected by an issue whereby the system may initially fail to resolve some DNS queries.
Subsequent queries for the same domain name, however, work as expected.
Only some domain names are affected.
Conditions:
- The BIG-IP system is configured with either a DNS Cache or Network DNS Resolver.
- The cache is still empty in regard to the domain name being resolved (for example, TMM has just started).
- The cache configuration specifies 'Use IPv6' (the default) but the system has no IPv6 default route.
Impact:
Initial DNS resolution of some domain names fails. Regardless of whether this happens via a DNS cache or Network DNS Resolver, the failure cascades to the client.
In the case of a DNS Cache, the client may just be returned with no record. In the case of a Network DNS Resolver, the failure will depend on the feature using the resolver.
For instance, SWG, SSLO, or the HTTP Explicit Forward Proxy, in general, are examples of features that rely on a Network DNS Resolver. In this case, the client's browser will fail to connect to the requested destination, and the client will be shown a proxy error.
Workaround:
Disable 'Use IPv6' in the affected DNS Cache or Network DNS Resolver.
You can keep the object in this state (with no consequences) until you define an IPv6 default route on the system, and you wish for the system to also use IPv6 to connect to Internet name-servers.
Component: Guided Configuration
Symptoms:
Attempting to install the AGC 8.0 results in the following error:
"Failed to load IApp artifacts from f5-iappslx-waf-bot-protection: java.lang.IllegalStateException: Failed to post template to block collection: ..."
Conditions:
The AGC is upgraded to a newer version via the "Access >> Guided Configuration" -> " Upgrade Guided Configuration".
Impact:
AGC 8.0 package is not installed.
Workaround:
Important: This workaround will delete all existing iApp configurations.
1. Download the latest AGC 8.0 package.
2. Open a new browser tab (first) and navigate to "Access >> Guided Configuration"
3. Open a new browser tab (second) and navigate to iApps >> Templates: Templates LX. Select all templates and click “Delete”.
4. In the second tab navigate to iApps >> Package Management LX. Select all existing packages and click “Uninstall”.
5. In the first tab click "Upgrade Guided Configuration" click and install the downloaded package.
Component: TMOS
Symptoms:
While attempting to view the properties for pool members via the Statistics page in the GUI, you see an error:
Instance not found: /Common/1234:80
Conditions:
-- A pool with at least one pool member with an IPv6 address.
-- Attempting to view the IPv6 pool member's properties via the Statistics page in the GUI.
Impact:
Unable to see the pool member's properties.
Workaround:
Use TMSH to view pool member properties:
# tmsh show ltm pool <pool name> members {<pool members>}
Component: TMOS
Symptoms:
Cannot update the firewall rule with 'any' value as the ip-protocol from the BIG-IP system GUI.
Conditions:
-- Create a rule and set protocol to TCP or UDP
-- From the GUI, change the protocol to "Any" and update
Impact:
Cannot update the firewall rule from GUI.
Component: TMOS
Symptoms:
The logintegrity script used to rotate the signature files for restnoded results in frequent cron errors similar to:
find: '14232restnoded_log_pattern': No such file or directory
Conditions:
Logintegrity is enabled.
Impact:
If logintegrity is enabled, the signature files for restnoded will not be in sync.
Workaround:
None.
Component: Global Traffic Manager (DNS)
Symptoms:
When you double-monitor a Generic Host Virtual Server (pool level + virtual server level) using the same SIP/HTTPS monitor, the 'Reason' is omitted from the output. 'tmsh show gtm server <server> virtual-servers' shows a "blank" reason for monitoring failure/success.
Conditions:
Double-monitor a Generic Host virtual server (pool level + virtual server level) using the same SIP/HTTPS monitor.
Impact:
Impedes your ability to identify the failure/success reason quickly.
Workaround:
Do not use the same monitor on both the virtual server and the pool level.
Component: TMOS
Symptoms:
When remote user authentication is configured, BIG-IP systems apply maximum active login token limitation of 100 to the default admin user account.
Conditions:
Remote Authentication is configured
Impact:
Unable to create more than 100 tokens for admin when remote authentication is configured
Component: TMOS
Symptoms:
After changing the management IP address via the GUI, the browser is not redirected, and reports Unable to connect BIG-IP device.
Conditions:
Change the Management IP address from the GUI and submit the change.
Impact:
Browser does not get redirected to the new address
Workaround:
Access the GUI by manually going to the new Management IP.
Component: Local Traffic Manager
Symptoms:
When a QUIT signal is sent over an FTP connection to an FTP virtual server during a data transfer, the data connection is closed immediately instead of waiting until the transfer is complete.
Conditions:
- BIG-IP configured with an FTP virtual server
- A client connects to the FTP virtual server
- Client starts an FTP data transfer
- Client sends a QUIT signal before the data transfer completes.
Impact:
FTP data connections are closed prematurely, causing incomplete data transfers.
Workaround:
This does not occur if the FTP profile for the FTP virtual server has inherit-parent-profile set to enable.
Component: TMOS
Symptoms:
In the GUI, if you filter the virtual server listing using an IP address, or part of an IP address, if there are any virtual servers that are using an address list that contains an address that matches that search string, those virtual servers will not show up in the search results.
Similarly, if you filter the virtual server listing using an IP address, or part of an IP address, if there are any virtual servers that are using an address that matches the search string, but are using a port list, those virtual servers will not show up in the search results.
Conditions:
-- Using Address Lists or Port lists with a virtual server.
-- Using the GUI to search for virtual servers based on address.
Impact:
Virtual servers that should match a search are not found.
Workaround:
None.
Component: TMOS
Symptoms:
TMM may prematurely shut down during its initialization when several TMMs and interfaces are configured. The system logs messages in one or more TMM log files (/var/log/tmm*):
MCP connection aborted, exiting.
Conditions:
-- BIG-IP Virtual Edition (VE).
-- Happens during TMM startup.
-- The issue is intermittent, but is more likely to occur on systems with a lot of TMMs (more 20 in most cases) and several interfaces (approximately 8 or more).
Impact:
TMM shuts down prematurely. Traffic disrupted while tmm restarts. Possible deadlock and MCP restart loop requiring a full reboot of the BIG-IP device.
Workaround:
None.
Component: Global Traffic Manager (DNS)
Symptoms:
Dangling monitor rule after pool deletion.
# tmsh delete gtm monitor tcp tcp_test
01070083:3: Monitor /Common/tcp_test is in use
Conditions:
Using transaction to delete pool and create pool of same name with different monitor.
Impact:
Unable to delete the remaining monitor.
Workaround:
1. # bigstart restart mcpd
2. Do not combine deletion and re-create pool in the same transaction.
Component: TMOS
Symptoms:
REST API calls will fail authentication for all users except an “admin” user.
Conditions:
REST API calls made by users other than “admin” user.
Impact:
REST API call will fail with a 401 code: “authorization failure” message.
Workaround:
For successful authentication, use an “admin” user for executing REST API calls.
Component: Global Traffic Manager (DNS)
Symptoms:
FastDNS packet processing might cause unexpected traffic drops.
Conditions:
-- FastDNS is in use.
The problem is more likely to occur on a systems with a low number of TMMs.
Impact:
Unexpected traffic drops
Component: TMOS
Symptoms:
When BGP graceful restart is configured for peers in IPv4 unicast and IPv6 unicast address families, after graceful restart for both IPv4 and Ipv6 address families, routes from IPv6 unicast address family might be missing.
Conditions:
- Different BGP peers configured in IPv4 unicast and IPv6 unicast address families.
- BGP graceful restart happens for both IPv4 unicast and IPv6 unicast.
Impact:
Routes from IPv6 peers are missing. They are also not present in the RIB.
Component: TMOS
Symptoms:
When creating a virtual server in the GUI with an IPv6 destination address and a port list (shared object), the system returns an error similar to:
"0107028f:3: The destination (0.0.0.0) address and mask (ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff) for virtual server (vs1) must be be the same type (IPv4 or IPv6)."
Conditions:
-- Creating or updating a virtual server.
-- Attempting to use an IPv6 Host address with a Port List shared object.
Impact:
Unable to create/modify virtual server.
Workaround:
Create an Address List shared object with the IPv6 address in it and use that instead of the Host address.
Component: TMOS
Symptoms:
While modifying a gateway-ICMP monitor you see the following error:
"01070374:3: Cannot modify the address type of monitor /Common/<monitor_name>."
Conditions:
-- Using the GUI to modify a Gateway-ICMP monitor
-- The monitor is attached with a pool that has one or more pool members.
Impact:
You cannot update the Gateway-ICMP monitor fields via the GUI.
Workaround:
Modify from tmsh using the following command: "tmsh modify ltm monitor gateway-icmp <monitor_name> [<field> <new_value>]"
Eg, To update description of a monitor named gw_icmp, use command: "modify ltm monitor gateway-icmp gw_icmp description new_description"
Component: TMOS
Symptoms:
In the GUI, while creating a virtual server with IPv6 address or port list in destination, BIG-IP should display the source address in IPv6 format rather than IPv4 format 0.0.0.0/0
Conditions:
This is encountered while creating a Virtual server with IPv6 address or port list in destination.
Impact:
An IPv6 address list is displayed in IPv4 notation
Component: Application Security Manager
Symptoms:
When the first DNS resolver is invalid, Bot Defense is unable to verify trusted bots, and is classifying them as malicious bots.
Conditions:
-- First DNS resolver in the list is invalid.
-- Bot Defense profile is attached to a virtual server.
-- Request from a search engine arrives.
Impact:
Bot Defense wrongly classifies valid search engines as malicious bots (and might block them if enforcement is enabled).
Workaround:
Fix the first DNS resolver in the list. It's possible that the first DNS resolver is the built in DNS resolver "f5-aws-dns".
Component: Application Security Manager
Symptoms:
Policy history files fill /var disk partition
Conditions:
Very large configuration with substantial history and config sync enabled
Impact:
ASM sync fails due to /var being full
Workaround:
The properties in '/etc/ts/tools/policy_history.cfg' file, for cleaning as a file-based configuration option, are a Support-oriented configuration option aimed to help in these cases.
Component: Access Policy Manager
Symptoms:
When attempting to access a resource that requires subsession validation, the client may receive an HTTP 503 "Service Unavailable" response, and the logs indicate that per-request policy execution time has expired.
Conditions:
Multiple connections are accessing the same subsession, triggering subsession lock contention.
Impact:
Some clients will fail to connect to their destination.
Workaround:
Add criteria to the gating criteria to enable more fine-grained subroutines to reduce subsession contention. For example, add category, or application name, to the gating criteria. In the case of API protection, consider concatenating credentials with the resource hostname (plus port).
Increase the per-request policy execution timeout value, controlled by the variable tmm.access.prp_global_timeout, to a higher value.
Component: TMOS
Symptoms:
Using "tmsh load sys config replace file ..." to change a traffic matching criteria's port type from port to port-list (or vice-versa) results in both types ending up in the traffic matching criteria object.
Conditions:
-- Using traffic-matching-criteria.
-- Using "tmsh load sys config replace file ..." or "tmsh load sys config merge file ..." to change the port to a port-list (or vice-versa).
Impact:
-- System does not load the desired configuration.
-- Virtual servers may match/process more traffic than expected.
Component: TMOS
Symptoms:
The tmrouted daemon restarts and produces a core file.
Conditions:
Exact trigger is unknown, but the issue was seen on a chassis setup during a blade failover
Impact:
Traffic disrupted while tmrouted restarts.
Workaround:
Component: Application Security Manager
Symptoms:
Some ASM sync files remain on disk after config sync finishes. They remain until periodic clean-up tasks activate
Conditions:
-- ASM provisioned
-- BIG-IP devices are in a sync group
-- Relatively small "/var" partition
Impact:
If the files are large it may lead to "lack of disk space" problem.
Component: TMOS
Symptoms:
When opening the GUI > Main > Statistics > Dashboard - the control plane CPU usage is around 7-15% on a completely empty system and Java consumes 3-5% CPU.
Conditions:
Leaving UI dashboard page left open.
Impact:
System performance is impacted if the dashboard page is kept open.
Component: TMOS
Symptoms:
In a device service cluster, changing a traffic-matching-criteria object's port configuration and then performing a full-sync will cause the sync target's traffic-matching-criteria ports to be modified incorrectly.
Once systems are in this state, further ConfigSyncs may result in these error messages:
err mcpd[6489]: 01070710:3: Database error (13), Cannot update_indexes/checkpoint DB object, class:traffic_matching_criteria_port_update status:13 - EdbCfgObj.cpp, line 127.
err mcpd[6489]: 01071488:3: Remote transaction for device group /Common/Failover to commit id 250 6869100131892804718 /Common/tmc-sync-2-bigip1.test 0 failed with error 01070710:3: Database error (13), Cannot update_indexes/checkpoint DB object, class:traffic_matching_criteria_port_update status:13 - EdbCfgObj.cpp, line 127..
Conditions:
-- Two or more BIG-IPs in a DSC.
-- Using traffic-matching-criteria, and making changes.
Impact:
BIG-IP configurations are out of sync (even though they show "In Sync"). Affected virtual servers will process more traffic than configured.
Workaround:
On an affected system, perform one of the two procedures to correct MCPD's in-memory configuration:
1. Remove the traffic-matching criteria from all virtual servers (or only affected virtual servers, if known), and then re-add the traffic-matching criteria.
2. Save the configuration and then follow the procedure in
K13030: Forcing the mcpd process to reload the BIG-IP configuration.
tmsh save sys config
clsh touch /service/mcpd/forceload
clsh reboot
Component: Application Security Manager
Symptoms:
Websocket URLs content profiles are not created as expected during JSON Policy import
Conditions:
Import JSON Policy with Websocket URLs configured with content profiles.
Impact:
Content profiles are not being added to the webscket URLs causing wrong configuration.
Workaround:
The content profiles can be manually associated after the import process using REST or GUI.
Component: Local Traffic Manager
Symptoms:
When the HTTP profile response-chunking option is set to "unchunk", chunked responses will be unchunked and the Transfer-Encoding header is removed.
If the server sends a chunked response with both Transfer-Encoding and Content-Length headers, Transfer-Encoding is removed but Content-Length is not. This causes the client to receive a response with an erroneous Content-Length header.
Conditions:
- HTTP virtual server with response-chunking set to "unchunk"
- Server response with both Transfer-Encoding and Content-Length headers present
Impact:
Malformed HTTP response received by client as length of response should be determined by closure of connection, not erroneous Content-Length header.
Workaround:
Implement iRule: at HTTP_RESPONSE time, remove the Content-Length header if the Transfer-Encoding header is also present.
Component: TMOS
Symptoms:
'table' command fails to resume causing processing of traffic to halt due to 'irule_scope_msg' causing iRule processing to proceed in a way that 'table' does not expect.
Conditions:
- iRule using 'table' command
- Diameter 'irule_scope_msg' enabled
Impact:
Traffic processing halts (no crash)
Component: TMOS
Symptoms:
Vlan information references a deleted partition on the primary device and configuration load fails giving an error:
emerg load_config_files[8487]: "/usr/bin/tmsh -n -g -a load sys config partitions all base " - failed. -- 01070277:3: The requested vlan, vlangroup or tunnel (/autoprov/pcs-autoprov-core-lb-link) was not found.
Conditions:
-- High availability (HA) environment
-- A VLAN is created in a partition on one device
-- The partition is deleted on the other device
Impact:
After config sync, the VLAN may point to the deleted partition. Configuration load will fail, and upgrade will also fail.
Workaround:
-- Remove the config from bigip_base.conf and load the config again (tmsh load sys config)
-- Run both create and delete for VLAN on both units and then only sync it (For both auto and manual sync)
Component: TMOS
Symptoms:
BIG-IP throws an error while upgrading:
The source (0.0.0.0%100) and destination (0.0.0.0) addresses for virtual server (/Common/test) must be in the same route domain.
Unexpected Error: Loading configuration process failed.
Conditions:
- non-default route-domain partition
- A virtual-address is using the same tmsh command that AS3 uses.
Example:
tmsh create net route-domain 100
tmsh create ltm virtual-address test address any%100
tmsh create ltm virtual test destination any%100:0
tmsh save sys config
Impact:
The configuration will not load when the device is rebooted, or when mcpd is restarted.
Component: Local Traffic Manager
Symptoms:
Modifying monitor properties gives error, if it is attached to a pool with Node/Pool member instance.
0107082c:3: Cannot modify the destination address of monitor /Common/my_monitor
Conditions:
-- Monitor with default properties.
-- Pool containing a node or pool member.
-- Monitor is attached to the pool.
Impact:
Monitor properties can't be modified if they are in use by a pool.
Workaround:
Remove monitor, modify it, and then add it back.
Component: TMOS
Symptoms:
When you try to configure an SNMP trap with an invalid network type, you get a confusing error:
01070911:3: The requested enumerated (aaa) is invalid (, mgmt, other) for network in /Common/snmpd trapsess (/Common/i2_2_2_1_1)
Conditions:
SNMP trap is configured with an invalid network enum type.
Impact:
The word 'none' is not shown in the error string.
Component: TMOS
Symptoms:
This issue can manifest in the following ways:
- Failure to restore a UCS archive to the currently active boot location (i.e. restoring a backup).
- Failure to restore a UCS archive to a different boot location by means of using the cpcfg utility (or the the "Install Configuration" option when changing boot locations in the Web UI).
- Failure to restore a UCS archive as part of a software upgrade (if rolling forward the configuration was requested, which is the default BIG-IP behavior).
In all cases, error messages similar to the following example are returned to the user:
/bin/sh: /bin/rm: Argument list too long
Fatal: executing: /bin/sh -c rm -fr /var/named/config/namedb/*
Operation aborted.
/var/tmp/configsync.spec: Error installing package
Config install aborted.
Unexpected Error: UCS loading process failed.
Conditions:
This issue occurs when a large number of DNS zone files are already present in the /var/named/config/namedb directory of the boot location to which the UCS archive is being restored.
Impact:
The UCS archive fails to restore. Additionally:
- If the UCS archive was being restored on the currently active boot location, the named and zrd daemons may not be running after the failure, leading to traffic outages.
- If the UCS archive was being restored as part of an upgrade, the installation will fail and the destination boot location will be marked as failed (thus preventing a BIG-IP Administrator from activating it).
Workaround:
Depending on the failure mode, perform one of the following workarounds:
- If you were restoring a UCS archive on the currently active boot location, run the following command, and then attempt the UCS archive restore operation again:
find /var/named/config/namedb -mindepth 1 -delete
- If you encountered the failure during an upgrade, it should mean you were installing an Engineering Hotfix (otherwise the /var/named/config/namedb directory on the destination boot location would have been empty).
Installing an Engineering Hotfix will actually perform two separate installations - first the base version, and then the hotfix on top of that. Each installation restores the source location's UCS archive.
The UCS installation performed during the base installation will work, and the one performed during the hotfix installation will fail (because DNS zone files are already in place now, and they will fail to be deleted).
In this case, you can work around the issue by performing two distinct installations (to the same destination boot location). First the base version by itself, and then the hotfix installation by itself:
Perform the first installation with the liveinstall.moveconfig and liveinstall.saveconfig db keys disabled. Perform the second installation after enabling the liveinstall.moveconfig and liveinstall.saveconfig db keys again.
- If you encountered the failure while using the cpcfg utility (or equivalent WebUI functionality), take a UCS archive instead, download it off of the BIG-IP or save it in a shared directory (e.g. /var/tmp), boot the system into the destination boot location, run the below command, and then restore the UCS archive:
find /var/named/config/namedb -mindepth 1 -delete
Component: Application Security Manager
Symptoms:
Asm_config_event_handler is consuming a lot of CPU while processing signatures after sync
Conditions:
This is encountered during a UCS load, or by a high availability (HA) configuration sync.
Impact:
Asm_config_server_rpc_handler.pl consumes excessive CPU and takes an exceedingly long time to complete.
Workaround:
Disable the signature staging action item for all policies.
Component: Local Traffic Manager
Symptoms:
When re-initializing an onboard HSM on particular platforms, BIG-IP may disconnect from the HSM after a second restart of the MCPD daemon.
This can occur when using administrative commands such as:
-- tmsh run util fips-util init
-- fipsutil init
-- tmsh run util fips-util loginreset -r
-- fipsutil loginreset -r
Conditions:
-- Using the following platforms:
+ i5820-DF / i7820-DF
+ 5250v-F / 7200v-F
+ 10200v-F
+ 10350v-F
+ vCMP guest on i5820-DF / i7820-DF
+ vCMP guest on 10350v-F
Impact:
BIG-IP is unable to communicate with the onboard HSM.
Workaround:
The last step in using "fipsutil init" is to restart all system services ("tmsh restart sys service all") or reboot.
Immediately before doing this:
-- open /config/bigip.conf in a text editor (e.g. vim or nano)
-- locate and delete the configuration "sys fipsuser f5cu" stanza, e.g.:
sys fipsuser f5cu {
password $M$Et$b3R0ZXJzCg==
Component: Advanced Firewall Manager
Symptoms:
Changing DOS vector enforcement configuration can cause BIG-IP to get stuck in hardware syncookie mode.
Conditions:
- Changing DOS vector enforcement configuration when device is in global syncookie mode.
For a list of platforms that support hardware syncookie protection, see https://support.f5.com/csp/article/K14779
Impact:
Device is stuck in hardware syncookie mode and generates syncookies.
Workaround:
Run the following command:
tmsh restart sys service tmm
Impact of workaround: restarting tmm disrupts traffic.
Component: Global Traffic Manager (DNS)
Symptoms:
TMM crashes and produces a core file.
Conditions:
The BIG-IP system configuration includes a Network DNS Resolver, which is referenced by another object (for example, a HTTP Explicit Forward Proxy profile) for DNS resolution.
Impact:
Traffic disrupted while tmm restarts.
Workaround:
You may be able to work around this issue by having the Network DNS Resolver work in forwarding/recursive mode rather than in resolving/iterative mode.
To do so, you configure a Forward Zone in the Network DNS Resolver for '.' (the DNS root). This causes DNS to send all DNS requests to a different, external resolver of your choice, which will perform recursive resolution.
The servers you configure for the '.' Forward Zone could be resolvers internal to your organization or public resolvers (e.g. Google DNS).
Component: Access Policy Manager
Symptoms:
The BIG-IP system sends cookies with no "domain" attribute set.
Conditions:
- Configure a Multidomain access profile
- Change the profile to single domain
Impact:
The BIG-IP system sends cookies with no "domain" attribute set. This causes the cookie to be not associated to a particular domain.
Workaround:
Manually remove the domain-groups config from the access profile by editing the bigip.conf.
Component: Service Provider
Symptoms:
In HTTP2 full proxy deployment, the LTM pool 'cur_sessions' statistics may show an unusually large number, such as 18446743927680663552
Conditions:
-- HTTP2 full proxy deployment
-- A client sends multiple requests over multiple streams
Impact:
'Current Sessions' statistics can be used to track number of pending requests in the queue and it can underflow.
Workaround:
None.
Component: Application Security Manager
Symptoms:
After upgrading BIG-IP, the Browser Challenges factory default update file cannot be installed, and you see this error:
Installation error: gpg: WARNING: unsafe ownership on homedir `/usr/share/live-update/share/gpg/browser_challenges_genesis_load'gpg: encrypted with 1024-bit ELG key, ID 7C3E3CE5, created 2007-03-20 "asm_sigfile_installer"gpg: Signature made Mon Aug 2
Conditions:
The new file that comes with the installation is ready to install
Impact:
New updated cannot be installed
Workaround:
There are 2 options:
1. download a new version of the update file (if exists)
2.1 download a copy of that file from the machine to your locale machine
2.2 rename it :
> cp BrowserChallenges_20200722_072935.im BrowserChallenges_<CURRENT_DATE>_<CURRENT_TIME>.im
## the date and time are for tracking and the have to be in a specific format DATE : YYYYMMDD, TIME: HHMMSS
2.3 upload the file and install it manually from the LiveUpdate screen.
Component: Application Security Manager
Symptoms:
Following Signature Update (ASM-SignatureFile_20200921_124008 or later), newly added/activated policies may fail Apply Policy due to a duplicate key database error:
----------------------------------------------------------------------
Sep 25 18:54:24 bigip1 crit g_server_rpc_handler_async.pl[16921]: 01310027:2: ASM subsystem error (asm_config_server.pl,F5::SetActive::Impl::set_active): Setting policy active failed: Failed to insert to DCC.ACCOUNT_NEGSIG_SIGNATURE_PROPERTIES (DBD::mysql::db do failed: Duplicate entry '8112518117000363265' for key 'PRIMARY' at /usr/local/share/perl5/F5/BatchInsert.pm line 219. )
----------------------------------------------------------------------
Conditions:
Signature Update ASM-SignatureFile_20200921_124008 is installed, and a newly imported or inactive policy is applied.
Impact:
Apply policy fails.
Workaround:
WORKAROUND 1:
- Install older signature update ASM-SignatureFile_20200917_175034
WORKAROUND 2:
- Disable staging for either signature 200101255 or signature 200101258 (or both) in the affected policies. The policy can then be successfully applied.
WORKAROUND 3:
- Run the following SQL to correct all affected policies on the device:
----------------------------------------------------------------------
UPDATE PL_POLICY_NEGSIG_SIGNATURES policy_sigs INNER JOIN (select previous_enforced_rule_md5, policy_id, count(*) as mycount from PL_POLICY_NEGSIG_SIGNATURES where previous_enforced_rule_md5 != '' group by previous_enforced_rule_md5, policy_id having mycount > 1) as multi_sigs on policy_sigs.policy_id = multi_sigs.policy_id and policy_sigs.previous_enforced_rule_md5 = multi_sigs.previous_enforced_rule_md5 SET policy_sigs.previous_enforced_rule_md5 = '', policy_sigs.previous_enforced_rule = '';
----------------------------------------------------------------------
Component: TMOS
Symptoms:
Despite having free memory, the BIG-IP system frequently logs kernel page allocation failures on B4450N blades to the /var/log/kern.log file like the following:
swapper/16: page allocation failure: order:2, mode:0x104020
After that, a stack trace follows. Note that the process name in the line ('swapper/16', in this example) varies. You may see generic Linux processes or processes specific to F5 in that line.
Conditions:
This occurs on B4450N blades regardless of configuration.
Impact:
As different processes can experience this issue, the system may behave unpredictably. For example, it is possible for a TMOS installation to fail as a result of this issue. Other processes may not exhibit any side effect as a result of this issue. The exact impact depends on which process becomes affected and how this process is designed to handle such a failure to allocate memory.
Workaround:
You must perform the workaround on each blade installed in the system.
-- If you want the workaround to survive reboots only, perform the following procedure:
1) Log on to the advanced shell (BASH) of the primary blade of the affected VIPRION system.
2) Run the following commands:
# clsh "sysctl -w vm.min_free_kbytes=131072"
# clsh "echo -e '\n# Workaround for ID950849' >> /etc/sysctl.conf"
# clsh "echo 'vm.min_free_kbytes = 131072' >> /etc/sysctl.conf"
-- If you want the workaround to survive reboots, upgrades, RMAs, etc., perform the following procedure:
1) Log on to the advanced shell (BASH) of the primary blade of the affected VIPRION system.
2) Run the following commands (with the desired amount in KB):
# clsh "sysctl -w vm.min_free_kbytes=131072"
# echo -e '\n# Workaround for ID950849' >> /config/startup
# echo 'sysctl -w vm.min_free_kbytes=131072' >> /config/startup
Note that the last two commands are not wrapped inside 'clsh' because the /config/startup file is already automatically synchronized across all blades.
Once the issue is fixed in a future BIG-IP version, remove the workarounds:
-- To remove the first workaround:
1) Edit the /etc/sysctl.conf file on all blades and remove the added lines at the bottom.
2) Reboot the system by running 'clsh reboot'. This will restore the min_free_kbytes kernel parameter to its default value for the BIG-IP version you are running.
-- To remove the second workaround:
1) Edit the /config/startup file on the primary blade only, and remove the extra lines at the bottom.
2) Reboot the system by running 'clsh reboot'. This restores the min_free_kbytes kernel parameter to its default value for the BIG-IP version you are running.
To verify the workaround is in place, run the following command (this should return the desired amount in KB):
# clsh "cat /proc/sys/vm/min_free_kbytes"
Component: TMOS
Symptoms:
Modifying a virtual server can cause BIG-IP to get stuck in hardware syncookie mode.
Conditions:
-- A virtual server is in hardware syncookie mode.
-- Modifying or deleting the virtual server
For a list of platforms that support hardware syncookie protection, see https://support.f5.com/csp/article/K14779
Impact:
Device is stuck in hardware syncookie mode and generates syncookies.
Workaround:
tmsh restart sys service tmm
Impact of workaround: restarting tmm disrupts traffic.
Component: Application Visibility and Reporting
Symptoms:
You cannot see reports (statistics->analytics->pool) when you choose to view by pool names.
Conditions:
This is encountered in the statistics screen.
Impact:
You can't see the statistics->analytics->pool report when you choose view by pool names.
Component: TMOS
Symptoms:
When BIG-IP Virtual Edition is running on GCP with mergeable buffers enabled, tmm might core while passing traffic. Subsequently, the kernel locks up which prevents the whole system from recovering.
Conditions:
- BIG-IP Virtual Edition running on GCP.
- Mergeable buffers (mrg_rxbuf) is enabled on the guest with direct descriptors.
Impact:
Traffic disrupted while tmm restarts.
Component: Local Traffic Manager
Symptoms:
HTTP does not close the TCP connection on the client if response is sent via HTTP::respond.
Conditions:
- TCP profile is used.
- HTTP profile is used.
- HTTP::respond iRule is used (via HTTP_RESPONSE_RELEASE).
- HTTP sends "Connection: close" header.
Impact:
TCP connection lives longer than needed.
Workaround:
Component: Access Policy Manager
Symptoms:
When user connects to the virtual server using Browser on their iPhone or Android device, webtop displays RDP resource. When they click on it, if SSO is not enabled, Remote Desktop Client App pops up with username pre-filled with string starting with f5_apm.
Conditions:
-- APM Webtop is configured with Single Sign-on disabled RDP resource.
-- Access the RDP resource from iOS or Android using RDP client.
Impact:
Remote Desktop Client App pops up with username pre-filled with string starting with f5_apm.
Workaround:
User needs to clear the username field and enter the actual username.
Component: Traffic Classification Engine
Symptoms:
Some blades return 'Unknown' for newly added URLs in a custom database
Conditions:
This might occur under when feedlist updates are done quickly in a loop
Impact:
BIG-IP is unable to classify using url categories defined in the custom database.
Workaround:
Restart wr_urldbd
Component: Local Traffic Manager
Symptoms:
When the QUIC PTO timer fires, it may resend some in-flight data. That data will not include any in-flight control frames.
Conditions:
A control frame is in-flight when the PTO timer fires.
Impact:
Minimal. The PTO timer is a mechanism to 'get ahead' of any lost packets and if a packet containing control frames is lost, those frames will be retransmitted.
Workaround:
None.
Component: Application Visibility and Reporting
Symptoms:
When upgrading to or loading saved configuration on BIG-IP software v13.0.0 or later, if the configuration contains AVR widgets created under a partition of '[All]', the config load fails.
Conditions:
This occurs if one or more AVR widgets in the configuration was created under the read-only '[All]' pseudo-partition.
This could have occurred if you were running a version of BIG-IP which did not include the fix for ID 721408.
Impact:
Upgrading to or loading an affected configuration on BIG-IP v13.x or later fails.
Workaround:
Manually edit the /config/bigip.conf configuration file and change '[All]' to 'Common':
# sed -i 's/\\\[All\]/Common/g' /config/bigip.conf
# tmsh load sys config
# tmsh save sys config
This should be done before upgrading to BIG-IP v13.x or later, or before saving configuration to be loaded later, or before loading a saved configuration from the config files.
Component: Access Policy Manager
Symptoms:
NTLM authentication fails with the error "RPC exception: Failed to verify checksum of the packet".
Conditions:
-- Start nlad process with "encryption".
-- Configure a user and map that user to a huge number of groups.
-- Configure NTLM front end authentication.
Impact:
User authentication fails
Workaround:
Run "nlad" process with "-encrypt no" in the file /etc/bigstart/startup/nlad.
Component: Access Policy Manager
Symptoms:
Client connections are reset and you see an error in /var/log/apm : "(ERR_NOT_FOUND) Category Lookup failed or a Category Lookup agent is not present in the policy before Response Analytics"
Conditions:
-- Category Lookup agent (lookup type SNI) in the per-request policy before Request or Response Analytics agent
-- Multiple requests sent in the same SSL connection.
Impact:
Connections are reset or they follow the fallback branch for subsequent requests in the same SSL connection
Component: Application Security Manager
Symptoms:
A false positive violation "Null in Request" is thrown erroneously.
Conditions:
-- BIG-IP receives a query string in the "Referrer" header
Impact:
False positive violation "Null in Request" is thrown
Workaround:
Component: Traffic Classification Engine
Symptoms:
Several new TLDs have been added and need to be classified. The classification results return "Unknown" when the new TLD is being queried.
Conditions:
New TLD is being queried
Impact:
The URL query with new TLDs can not be blocked with custom feed list.
Custom, Webroot, and Cloud returns Unknown category.
Workaround:
Configure CPM policy to classify traffic based on hostname or SNI rather than urlcat.
Component: Performance
Symptoms:
- TMM crashes
- kernel panics
- BIG-IP core file created
- Cloud Failover Extension unexpected behavior (where applicable)
Conditions:
- BIG-IP Azure Virtual Edition
- Azure Host performs a Network Management Agent (NMAgent) update while TMM is running
- BIG-IP VE using Accelerated Networking
Impact:
- Traffic disrupted while tmm restarts
- BIG-IP restarts
- Cloud Failover Extension state data lost (where applicable)
Workaround:
- Disable Accelerated Networking on BIG-IP network interfaces (Reversed settings from Azure documentation)
Individual VMs & VMs in an availability set
First stop/deallocate the VM or, if an Availability Set, all the VMs in the Set:
Azure CLI
az vm deallocate \
--resource-group myResourceGroup \
--name myVM
Important, please note, if your VM was created individually, without an availability set, you only need to stop/deallocate
the individual VM to disable Accelerated Networking. If your VM was created with an availability set, all VMs contained in
the availability set will need to be stopped/deallocated before disabling Accelerated Networking on any of the NICs.
Once stopped, disable Accelerated Networking on the NIC of your VM:
Azure CLI
az network nic update \
--name myNic \
--resource-group myResourceGroup \
--accelerated-networking false
Restart your VM or, if in an Availability Set, all the VMs in the Set and confirm that Accelerated Networking is disabled:
Azure CLI
az vm start --resource-group myResourceGroup \
--name myVM
Component: TMOS
Symptoms:
In case of traffic selector narrow-down, SA's do not come up after reboot on standby.
Conditions:
A traffic-selector mismatch occurs between initiator and responder.
Impact:
Tunnel will not come up on standby.
Workaround:
Configure exactly matching traffic selector on both initiator and responder.
Component: Local Traffic Manager
Symptoms:
The BIG-IP system can redirect a request to a fallback host when the target pool is unavailable. If the configured fallback host within the HTTP profile contains HTTP iRule commands such as HTTP::host or HTTP::uri, the corresponding HTTP request can fail. A connection reset may be encountered instead.
Conditions:
- HTTP profile with "fallback-host" profile option configured containing an HTTP iRule command.
Impact:
When utilizing the HTTP profile with "fallback-host" profile option with HTTP iRule commands, incorrect connection resets can be seen by the client instead of the correct HTTP response.
Workaround:
Attaching an iRule containing HTTP::redirect or similar command to the virtual server can be used instead of the fallback host-profile option to redirect traffic to another virtual server.
Component: TMOS
Symptoms:
Pam-authenticator cores. There is a log message in /var/log/user/log:
err pam-authenticator: tac_author_read: short author header, -1 of 12: Connection reset by peer
Conditions:
-- TACACS auth configured on BIG-IP
-- A BIG-IP user authenticates and the user is a TACACS user
-- The TACACS server resets the connection.
Impact:
Pam-authenticator fails with segfault or sigabrt, and the user is unable to authenticate to BIG-IP.
Component: Local Traffic Manager
Symptoms:
Tcp proxy does not ignore HUDEVT_CHILD_CONNECTED and gives an error message:
hud_tcp_serverside_handler/3676: 6.0.0.20.21 - 6.6.6.1.51147: unexpected serverside message HUDEVT_CHILD_CONNECTE
Conditions:
FTP profile is in use
Impact:
Logs entries in the log file
Workaround:
Component: TMOS
Symptoms:
When a large number of virtual servers use the same ASM policy from a manually-created LTM Traffic policy, the Security tab of the virtual server takes a long time to load.
Conditions:
Large number of virtual servers using the same ASM policy
Impact:
Loading of Security tab of a virtual server takes a long time
Workaround:
Component: Application Security Manager
Symptoms:
BD sometimes crashes when handling HTTP traffic
Conditions:
High memory utilization
Impact:
Traffic disrupted while bd restarts.
Workaround:
Component: Application Security Manager
Symptoms:
1) var/lib/mysql/mysqld.err is filled with log entries containing:
------------
200824 11:04:43 [ERROR] mysqld: Can't open file: './PRX/REQUEST_LOG.frm' (errno: 24)
200824 11:18:46 [ERROR] mysqld: Can't open file: './PRX/REQUEST_LOG.frm' (errno: 24)
200824 11:35:58 [ERROR] mysqld: Can't open file: './PRX/REQUEST_LOG.frm' (errno: 24)
------------
2) There are a lot of PRX.REQUEST_LOG partitions, in some cases up to 1024, many of which are empty.
Conditions:
ASM/AVR provisioned
Impact:
Mysql out of resources when opening files
PRX.REQUEST_LOG Corrupt
Workaround:
0) in case /appdata partition is filled up to 100% and mysql restarts continuously -
https://support.f5.com/csp/article/K14956
https://support.f5.com/csp/article/K42497314
1) Look into -
SELECT * FROM INFORMATION_SCHEMA.PARTITIONS WHERE table_name = 'REQUEST_LOG' AND table_schema = 'PRX'\G
to identify all the empty partitions.
2) Manually (or via shell script) execute this sql -
ALTER TABLE PRX.REQUEST_LOG DROP PARTITION empty_partition_name
where "empty_partition_name" is the partition name as "p100001" for every partition that is empty.
4) increase 'open_files_limit' to '10000'.
--------------------------------
In /etc/my.cnf file:
1. Change the value of the 'open_files_limit' parameter to 10000.
2. Restart MySQL:
bigstart restart mysql
--------------------------------
5) pkill asmlogd
Note: This workaround does not survive upgrade. It must be reapplied after every upgrade until the upgraded version contains a fix.
Component: Global Traffic Manager (DNS)
Symptoms:
GTM is unable to load the configuration.
Conditions:
-- GTM has been upgraded to a version with fix for ID722682 from a version that does not have the fix for ID722682
-- A GTM server has a name with no colon
-- That GTM server has a virtual server with colon in the name
-- That virtual server is added to a pool
Impact:
GTM config file cannot be loaded successfully after upgrade.
Workaround:
Edit bigip_gtm.conf manually to delete "\\" or replace colon ":" with other non-reserved char. such as "-".
Component: Local Traffic Manager
Symptoms:
Unable to delete monitor with an error similar to:
01070083:3: Monitor /Common/my-mon is in use.
Conditions:
-- HTTP monitors are attached directly to pool members, or node-level monitors exist.
-- Performing an operation that causes the configuration to get rebuilt implicitly, such as "reloadlic".
Impact:
Unable to delete object(s) no longer in use.
Workaround:
When the system gets into this state, save and reload the configuration:
tmsh save sys config && tmsh load sys config
Component: Local Traffic Manager
Symptoms:
HTTP::close used in iRule might not close connection. For example:
when HTTP_REQUEST {
HTTP::close
HTTP::respond 200 -version 1.1 content "OK" Content-Type text/plain
Conditions:
Using HTTP::close along with HTTP::respond
Impact:
HTTP connection can be re-used.
Workaround:
Explicitly add close header in the HTTP::respond. For example:
HTTP::respond 200 content "OK" Connection close
Component: TMOS
Symptoms:
The 'tmsh run sys integrity status-check -a -v' or 'tpm-status' commands incorrectly report system integrity status as 'Invalid' even when the system software has not been modified.
Conditions:
This occurs if all of the following conditions are true:
-- BIG-IP software v14.1.0 or later version.
-- Engineering Hotfix installed on TPM-supported BIG-IP platform.
-- The Engineering Hotfix contains a fix for ID893885
-- The Engineering Hotfix contains an updated 'sirr-tmos' package
Impact:
Incorrect presentation of system software status.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
A TLS 1.3 handshake failure occurs when using openssl's AES-GCM cipher in FIPS mode.
Conditions:
FIPS mode and attempting TLS 1.3 with cipher AES-GCM
Impact:
Handshake failure for TLS 1.3
Workaround:
Disable FIPS mode, or alternately use non AES-GCM cipher for TLS 1.3.
Component: TMOS
Symptoms:
When accessing the iApp Components tab, the system presents an error similar to the following:
An error has occurred while trying to process your request.
Conditions:
-- Upgrade to v15.1.x.
-- Change partition
-- Navigate to GUI iApps ›› Application Services : Applications, try to view any custom iApp.
Impact:
Unable to view/modify iApps via GUI iApps ›› Application Services : Applications screen.
Workaround:
To reconfigure the iApp, do the following:
1. Navigate to the following location in the GUI:
Local Traffic :: Virtual Server List.
2. Click the Application Link :: Reconfigure.
Note: Although this allows you to reconfigure an iApp, it does not provide access to the iApp application details Components page.
Component: Access Policy Manager
Symptoms:
End users are unable to access an application even though the active tokens are far less than allowed limit, with this error:
/Common/my_oauth:Common: Request Access Token from Source ID <id> IP <ip> failed. Error Code (access_denied) Error Description (This user has reached configured access token limit.)
Conditions:
1. configure per user access token limit
2. revoke some tokens
3. restart tmm
Impact:
User is denied access even though token limit per user is not reached
Component: TMOS
Symptoms:
Whenever snmp user is added with short password (less than 8 characters), tmsh allows this during creation but throws an error when snmpwalk is done. But in GUI it throws error "Password must have at least 8 characters." during creation itself.
Conditions:
Configuring snmp user from tmsh with password less than 8 characters leads to this problem of getting error during snmp walk.
Impact:
snmpwalk fails
Workaround:
Configure snmp user with password greater than or equal to 8 characters.
Component: TMOS
Symptoms:
BIG-IP might transmit excessive multicast/broadcast traffic.
Conditions:
-- BIG-IP Virtual Edition with more than one TMM.
-- Number of excessive packets is directly proportional to the number of TMMs.
Impact:
Excessive multicast/broadcast traffic.
Component: Application Security Manager
Symptoms:
When getcrc tool displays help to the end user, it displays a directory structure instead of version.
Conditions:
Displaying help in getcrc utility.
Impact:
Version information is not displayed.
Component: Local Traffic Manager
Symptoms:
When an LTM policy is applied to HTTP/2 traffic and refers to TCL expression(s), TMM may crash.
Conditions:
-- A virtual is configured with http and http2 profiles.
-- An LTM policy is published and refers to TCL expression(s).
-- The policy is attached to the virtual server.
Impact:
Traffic disrupted while tmm restarts.
Component: TMOS
Symptoms:
- TMM is stuck restarting on a bigip device or VE instance
- Unable to process application traffic
- /var/log/tmm logfile may contain the notice "MCP connection aborted, exiting"
Conditions:
- Remote logging destination is configured (System-Logs-Configuration-Remote logging) and pointing to a FQDN hostname
- FQDN hostname temporarily or permanently fails to resolve (e.g. DNS failure or congestion)
- Reboot or upgrade of a bigip instance or other conditions leading to tmm restart
Impact:
- Unable to process application traffic since the required component cannot be brought up
Workaround:
- Perform audit of DNS server(s), make sure FQDN destination resolves successfully, OR
- Use IP address as remote logging destination, OR
- Disable remote logging feature
Component: Application Security Manager
Symptoms:
When "Null in multi-part" violation is detected, BIG-IP logs it to remote log even when the violation is not set to blocked in Learning and Blocking settings
Conditions:
This happens when "Null in multi-part" violation is detected
Impact:
Incorrect logging to remote logs
Workaround:
Component: Application Security Manager
Symptoms:
Live update is not working when BIG-IP DNS is configured to use IPv6
Conditions:
BIG-IP DNS uses IPv6
Impact:
-- Unable to install latest updates to signatures.
-- Unable to import user-defined signatures.
Workaround:
If possible, use IPv4 for DNS.
Component: Local Traffic Manager
Symptoms:
An incorrect LTM policy rule is picked up e.g. a rule which should match first is omited.
Conditions:
Policy contains multiple rules which employ TCP address matching condition.
Impact:
Inocorrect LTM policy is applied.
Component: TMOS
Symptoms:
BIG-IP systems in a device group do not stay in sync if config sync is manual and constantly syncs if config sync is automatic.
The BIG-IP system constantly downloads the certificate bundle if the CA-bundle manager config includes a URL.
Conditions:
The CA-bundle manager is configured.
Impact:
The keymgmtd and mcpd process gets into a loop that causes constant config changes and if the ca-bundle-manager includes a URL, the BIG-IP system constantly downloads the bundle.
Component: TMOS
Symptoms:
If a BGP neighbor is configured with 'default originate,' the nexthop advertised for the default route may be incorrect.
Conditions:
-- Dynamic routing enabled.
-- Using BGP.
-- BGP neighbor configured with 'default originate'.
Impact:
The default route advertised via BGP is not acceptable to peers until the BGP session is cleared.
Workaround:
In imish, run the command:
clear ip bgp <affected neighbor address>
Component: Local Traffic Manager
Symptoms:
After upgrade, the 'DEFAULT' cipher in the serverssl profile attached to the HTTPS monitor does not include "ECDHE-RSA-AES256-CBC-SHA" cipher suite in the Client Hello.
Conditions:
After upgrade, HTTPS monitor cipherlist is read from serverssl profile ciphers and set to DEFAULT after upgrade.
Impact:
Upgrade breaks the SSL pool monitoring.
Workaround:
Set ciphers as DEFAULT:+SHA:+3DES:+EDH for profile server-ssl
Component: Local Traffic Manager
Symptoms:
If the SETTINGS_MAX_CONCURRENT_STREAMS setting is exceeded, BIG-IP sends a GOAWAY frame; however, browsers expect a RST_STREAM and the GOAWAY frame results in a half-rendered web page.
Conditions:
The maximum streams setting is exceeded on a HTTP/2 connection.
Impact:
BIG-IP sends a GOAWAY frame, and the browser shows a half-rendered page.
Workaround:
None.
Component: TMOS
Symptoms:
License activation http/https request has the license server IP address instead of license server domain name.
Conditions:
When the proxy server and proxy port are configured and delete default management route.
Impact:
This causes your proxy server to disallow the connection
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
In SSL reverse proxy, dynamic CRL checking for client certificate is not working when TLS 1.3 handshake is used.
The SSL handshake successfully completed even though the client certificate is revoked.
Conditions:
-- Dynamic CRL checking enabled on a client-ssl profile
-- The client-side SSL handshake uses TLS1.3.
Impact:
The handshake should fail but complete successfully
Component: Local Traffic Manager
Symptoms:
The SSL monitor remains in the current TLS version and does not switch to another version when a server changes.
Conditions:
-- SSL monitor configured.
-- Server configuration changes from TLSv1.2 to TLSv1.
Impact:
Pool members marked down.
Workaround:
Use the In-TMM monitor.
Component: Local Traffic Manager
Symptoms:
In-tmm https monitors do not send the SNI (Server Name Indication) information for non-default route domain pool members.
Conditions:
-- SNI is configured in serverssl profile a
-- serverssl profile is assigned to in-tmm https monitors
-- https monitors are monitoring pool members that are in a non-default route domain.
Impact:
The TLS connection might fail.
Workaround:
Component: Access Policy Manager
Symptoms:
When an Access Policy is configured with Maximum Session Timeout, the rendered value of maximum remaining session's time can flip/flop in seconds on a user's webtop
Conditions:
Access Policy is configured with Maximum Session Timeout >= 60000 secs
Impact:
End users will see the remaining time being continually reset.
Component: Access Policy Manager
Symptoms:
In case of Multi-Factor Authentication (MFA), the challenge response is handled by the 401 response agent, in the following scenario:
-- The policy does not have logon page agent.
-- AAA/Radius server action is configured to send challenge response to the client.
-- The policy has a 401 response logon agent.
The APM end user client sees the credentials popup for 401 response.
After entering and sending credentials (challenge response code received from RSA server via email or SMS), the Access Policy action fails.
Conditions:
-- Challenge response via RSA server or other.
-- Logon Page logon agent not implemented.
Impact:
Unable to use challenge response sent by RSA (or AAA RADIUS server).
Workaround:
Component: Local Traffic Manager
Symptoms:
When HTTP::respond is used in ADAPT_REQUEST_RESULT in iRule event, an error message "http_process_state_prepend - Invalid action:0x10a071 clientside" is observed in /var/log/ltm log.
Since HTTP::respond is not supported in ADAPT_REQUEST_RESULT, a better tcl error should be shown in /var/log/ltm like "TCL error: /Common/AVScan_Content <ADAPT_REQUEST_RESULT > - Illegal argument. Can't execute in the current context"
Conditions:
This can occur when HTTP::respond is used in ADAPT_REQUEST_RESULT:
when ADAPT_REQUEST_RESULT {
if {[ADAPT::result] contains "respond"} {
HTTP::respond 403 -version auto ...
Impact:
TCL error messages don't give adequate information to explain what triggered the TCL error.
Component: TMOS
Symptoms:
Neurond continuously restarts.
Conditions:
-- BIG-IP iSeries hardware platform
-- issuing the command "service --status-all"
Impact:
Neuron communications will be impacted
Component: TMOS
Symptoms:
After changing an element of the ike-peer configuration, such as the pre-shared secret, the related IPsec interface mode tunnel can no longer forward packets into the tunnel.
Conditions:
-- IKEv1.
-- IPsec policy in interface mode.
-- Configuration change or a new configuration.
Impact:
When the problem is exhibited on a BIG-IP Initiator, ISAKMP negotiation fails to start when interesting traffic arrives. No messages are written to /var/log/racoon.log.
When the problem is exhibited on a BIG-IP Responder, the IPsec tunnel successfully establishes, but the BIG-IP fails to forward any interesting traffic into the established tunnel. No useful log messages are seen.
Note: 'Interesting traffic' is packets that match a traffic-selector.
Workaround:
Run 'bigstart restart tmm' or reboot.
Component: TMOS
Symptoms:
On the dashboard, the line chart of 'Throughput', 'CPU Usage', and 'Memory Usage' can show the line of thresholds that indicates 'Upper Bound' and 'Lower Bound', but there is no line of thresholds on the line chart of 'Connections'.
Conditions:
Create a custom line chart for 'Connections' that includes 'Upper Bound' and 'Lower Bound'.
Impact:
Line Chart of 'Connections' does not display 'Upper Bound' and 'Lower Bound' thresholds.
Workaround:
None.
Component: TMOS
Symptoms:
Performing a full configuration sync with traffic-matching-criteria (TMC) under specific conditions fails with errors similar to:
err mcpd[6489]: 01070710:3: Database error (13), Cannot update_indexes/checkpoint DB object, class:traffic_matching_criteria_port_update status:13 - EdbCfgObj.cpp, line 127.
err mcpd[6489]: 01071488:3: Remote transaction for device group /Common/Failover to commit id 245 6869100131892804717 /Common/tmc-sync-2-bigip1.test 0 failed with error 01070710:3: Database error (13), Cannot update_indexes/checkpoint DB object, class:traffic_matching_criteria_port_update status:13 - EdbCfgObj.cpp, line 127..
Conditions:
This may occur on a full-load config sync (not an incremental sync)
On the device receiving the ConfigSync:
- a traffic-matching-criteria is attached to a virtual server
- the traffic-matching-criteria is using a port-list
On the device sourcing the ConfigSync:
- the same traffic-matching-criteria is attached to the same virtual server
- the original port-list is modified (e.g. a description is changed)
- the TMC is changed to reference a _different_ port-list
Impact:
Unable to sync configurations.
Workaround:
Copy the "net port-list" and "ltm traffic-matching-criteria" objects from the source to target system, merge them with "tmsh load sys config merge", and then perform a force-full-load-push sync from source to target.
If the BIG-IP systems are using device groups with auto-sync enabled, disable auto-sync temporarily while performing this workaround.
1. On the source system (the system whose configuration you want to sync to peer), save the configuration and extract the ltm traffic-matching-criteria and port-lists:
tmsh save sys config
(shopt -s nullglob; echo "#"; echo "# $HOSTNAME"; echo "# generated $(date +"%F %T %z")"
cat /config{/partitions/*,}/bigip{_base,}.conf |
awk '
BEGIN { p=0 }
/^(ltm traffic-matching-criteria|net port-list) / { p=1 }
/^}/ { if (p) { p=0; print } }
{ if (p) print; }
' ) > /var/tmp/portlists-and-tmcs.txt
2. Copy /var/tmp/portlists-and-tmcs.txt to the target system
3. On the target system, load that file:
tmsh load sys config replace file /var/tmp/portlists-and-tmcs.txt
3a. If loading the config file on the target system fails with the same error message seen during a ConfigSync, follow the procedure in
K13030: Forcing the mcpd process to reload the BIG-IP configuration.
tmsh save sys config
clsh touch /service/mcpd/forceload
clsh reboot
4. On the source system, force a full-load sync to the device-group:
tmsh run cm config-sync force-full-load-push to-group <name of sync-group>
Component: TMOS
Symptoms:
While trying to use the monitor test page in the GUI for an LDAP monitor, the destination address field contains the value *.*(asterisk-dot-asterisk), and it is not editable.
Conditions:
-- Create an LDAP monitor.
-- Go to the Test page.
Impact:
You are unable to modify the test destination address for the LDAP monitor.
Workaround:
Use TMSH.
-- To run the test:
tmsh run /ltm monitor ldap <LDAP-Monitor-Name> destination <IP-Address>
Example: tmsh run /ltm monitor ldap myldap destination 10.10.10.10:389
-- To check test results:
tmsh show /ltm monitor ldap <LDAP-Monitor-Name> test-result
tmsh show /ltm monitor ldap myldap test-result
Component: Application Security Manager
Symptoms:
Verification may be incomplete when using the F5 Anti-Bot Mobile SDK with the Bot Defense profile.
Conditions:
-- Using the Bot Defense profile together with the F5 Anti-Bot Mobile SDK.
-- Enabling the Mobile Applications section in the profile.
Impact:
Mobile application verification may be incomplete.
Workaround:
Component: TMOS
Symptoms:
When bulk deleting a large number of Bot Defense profiles (around 450 profiles) using TMSH, mcpd could crash.
Conditions:
This can be encountered during bulk delete of Bot Defenese profiles via tmsh.
Impact:
Crash of mcpd causing failover.
Workaround:
Delete the Bot Defense profiles in smaller batches to avoid the possible crash.
Component: TMOS
Symptoms:
When using Ansible to create a pool that contains an IPv6 pool member, you get an error:
0107003a:3: Pool member node and existing node cannot use the same IP Address.
Conditions:
-- Creating a new pool via Ansible.
-- A new IPv6 pool member is used.
-- The IPv6 pool member's name is not included.
Impact:
Pool creation fails.
Workaround:
If you are unable to specify the pool member name, you can use other available configuration tools like tmsh, the GUI, or AS3.
Component: Access Policy Manager
Symptoms:
PRP LDAP Group Lookup agent, the expression incorrectly places the 'string tolower' outside the square brackets. This causes an issue in the GUI of the LDAP Group Lookup object where the 'Simple' branch rules do not show up. You see this 'Warning':
Warning, this expression was made manually and couldn't be parsed, please use advanced tab.
Conditions:
Configure PRP with the LDAP Group Lookup agent in the Visual Policy Editor (VPE).
Impact:
Tcl expression containing a syntax error prevents the LDAP Group Lookup agent from functioning properly.
Workaround:
Go to the LDAP Group Lookup agent advanced tab and change this:
expr {[string tolower [mcget {session.ldap.last.attr.memberOf}]] contains string tolower["CN=MY_GROUP, CN=USERS, CN=MY_DOMAIN"]}
To this:
expr {[string tolower [mcget {session.ldap.last.attr.memberOf}]] contains [string tolower "CN=MY_GROUP, CN=USERS, CN=MY_DOMAIN"]}
Click finish.
Now you can click 'change', and use the 'Simple' tab and the 'Add an expression using presets' option.
Component: Local Traffic Manager
Symptoms:
When an SMTPS profile is applied on a virtual server and the SMTP client sends a STARTTLS command containing trailing white space, the BIG-IP system replies with '501 Syntax error'. The command is then forwarded to the pool member, which can result in multiple error messages being sent to the SMTP client.
Conditions:
-- A virtual server is configured with an SMTPS profile.
-- The SMTP client sends a STARTTLS command with trailing spaces.
Impact:
The SMTP client is unable to connect to the SMTP server.
Workaround:
Use an SMTP client that does not send a command containing trailing white space.
Component: Access Policy Manager
Symptoms:
Export of Access Policy times out after five minutes and fails via the GUI
Conditions:
Exact conditions are unknown, but it occurs while handling a large access policy.
Impact:
Unable to redeploy configuration
Workaround:
Use thge cli to export the access policy:
K30575107: Overview of the ng_export, ng_import and ng_profile commands
https://support.f5.com/csp/article/K30575107
Component: Advanced Firewall Manager
Symptoms:
Inconsistent configuration checks may result in configuration not being applied correctly.
Conditions:
DoS vectors configured on the BIG-IP
Impact:
Configuration changes may not be accepted by the BIG-IP.
Component: TMOS
Symptoms:
During boot of a i15xxx system you see the message:
Dataplane INOPERABLE - Only 7 HSBs found. Expected 8
Conditions:
There are no specific conditions that cause the failure.
This can occur on any i15xxx device, although some devices exhibit the failure consistently around 50% of boots and others never exhibit the issue.
Impact:
When this failure occurs in a system, the system is inoperable.
Workaround:
There is no workaround for systems that do not have software capable of resetting the hardware device during the HSB load process.
Component: Device Management
Symptoms:
You cannot upload a cert/key via the REST API if you are using a certificate manager account
Conditions:
-- Using the REST API to upload a certificate and/or key
-- User is logged in as a Certificate Manager
Impact:
Unable to upload certificates as Certificate Manager
Workaround:
Use admin account instead of using Certificate Manager account to upload certs and keys
Component: TMOS
Symptoms:
Installation of declarative onboarding RPM fails.
Conditions:
Use of icontrollx_package_urls in tmos_declared block to download/install RPMs via a URL.
Impact:
RPMs cannot be downloaded for declarative onboarding where RPMs are referenced via URL.
Workaround:
RPMs must be installed manually.
Component: Local Traffic Manager
Symptoms:
With certain configurations, virtual server keeps rejecting connections for rstcause 'VIP down' after 'trigger' events.
Conditions:
Required Configuration:
-- On the virtual server, the service-down-immediate-action is set to 'reset' or 'drop'.
-- The pool member has rate-limit enabled.
Required Conditions:
-- Monitor flap, or adding/removing monitor or configuration change made with service-down-immediate-action.
-- At that time, one of the above events occur, the pool member's rate-limit is active.
Impact:
Virtual server keeps rejecting connections.
Workaround:
Delete one of the conditions.
Note: The affected virtual server may automatically recover upon the subsequent monitor flap, etc., if no rate-limit is activated at that time.
Component: Local Traffic Manager
Symptoms:
Persistence records accumulate over time due to expiration process not reliably taking effect. The 'persist' memory type grows over time.
Conditions:
-- Non-cookie, non-mirrored persistence configured.
-- No high availability (HA) configured or HA connection permanently down.
-- Traffic that activates persistence is occurring.
Impact:
Memory pressure eventually impacts servicing of traffic in a variety of ways. Aggressive sweeper runs and terminates active connections. TMM may restart. Traffic disrupted while tmm restarts.
Workaround:
Component: Application Security Manager
Symptoms:
When server respond with a redirect, ASM challenge makes Google Analytics stats appear as if they are 'Direct' instead of 'Organic'.
Conditions:
-- Google link is responded to (by the server) with a redirect.
-- Bot defense profile or DoS Application profile attached to a virtual server with challenge mitigation enabled.
Impact:
Incorrect data is displayed in the Google Analytics dashboard.
Workaround:
Component: TMOS
Symptoms:
When performance tests are run on BIG-IP Virtual Edition (VE) in Microsoft Azure, the BIG-IP system loses all connectivity to the pools, virtual servers, and management address. It remains unresponsive until it is rebooted from the Azure console.
Conditions:
Running performance tests of VE in Azure.
Impact:
The GUI becomes unresponsive during performance testing. VE is unusable and must be rebooted from the Azure console.
Workaround:
Reboot from the Azure console to restore functionality.
Component: Local Traffic Manager
Symptoms:
Certain long-running iRule LX nodejs processes consume considerably more memory on the secondary blade than on the primary blade of the same cluster. The problem is present only on the Standby BIG-IP. On the Active the restjavad processes consume the same amount of memory on the primary and on the secondary slot.
Conditions:
-- VIPRION with multiple blades and vCMP configured
-- iRulesLX in use
Impact:
Higher memory consumption on the secondary blades, but the memory consumption does not increase, so the impact to system performance is negligible.
Component: Local Traffic Manager
Symptoms:
Occasionally the Nitrox3 ZIP engine hangs.
In /var/log/ltm:
crit tmm[12404]: 01010025:2: Device error: n3-compress0 Nitrox 3, Hang Detected: compression device was reset (pci 02:00.1, discarded 1).
crit tmm[12404]: 01010025:2: Device error: n3-compress0 Zip engine ctx eviction (comp_code=0): ctx dropped.
Conditions:
BIG-IP appliance that uses the Nitrox 3 hardware compression chip: 5xxx, 7xxx, 12250 and B2250.
You can check if your platform has the nitrox3 by running the following command:
tmctl -w 200 compress -s provider
provider
--------
bzip2
nitrox3 <--------
Impact:
The Nitrox3 hardware compression system becomes unavailable and the compression mode switches to software compression. This can lead to high CPU usage.
Workaround:
Disable http compression
Component: Application Security Manager
Symptoms:
The name provided by getcrc tool provides incorrect ASM cookie name when cookie attributes path or/and domain is/are present in response from server
Conditions:
This is applicable when domain and path cookie attributes are present in response from server
Impact:
ASM cookie name which is displayed is incorrect
Workaround:
Component: Policy Enforcement Manager
Symptoms:
Flows for a PEM subscriber are not deleted from the system even after the subscriber is deleted.
Conditions:
When IPv6 prefix flows are configured on PEM (i.e., sys db variable tmm.pem.session.ipv6.prefix.len is configured with a value other than 128).
Impact:
Flows for a PEM subscriber are not deleted from the system even after the subscriber is deleted. Resources are not released from the system.
Workaround:
None.
Component: TMOS
Symptoms:
The Mellanox CX5 Ex adapter is not supported by the BIG-IP.
Conditions:
A BIG-IP Virtual Edition system configured to use one or more Mellanox CX5 Ex adapters.
Impact:
Systems using a CX5 Ex adapter will have to use the sock driver rather than the Mellanox driver.
Component: Local Traffic Manager
Symptoms:
The node iRule command causes the specified server node to be used directly, thus bypassing any load-balancing. However, with HTTP/2, the node command may fail to execute within the CLIENT_ACCEPTED event. This results in no traffic being sent to configured node.
Conditions:
-- A node command is used under CLIENT_ACCEPTED event.
-- An HTTP/2 profile applied to virtual server.
-- The HTTP/2 protocol in use.
Impact:
With HTTP/2 configured, the iRule node command fails to execute within the CLIENT_ACCEPTED event, causing no traffic to be sent to the desired node.
Workaround:
As a workaround, you may use HTTP_REQUEST event instead of CLIENT_ACCEPTED in iRule syntax.
Component: Global Traffic Manager (DNS)
Symptoms:
After upgrading a FIPS-enabled BIG-IP system, booting to a partition running an earlier software version, which results in a libcrypto validation error and system halt.
Conditions:
-- FIPS-licensed BIG-IP system.
-- Upgrade.
-- Boot into an partition running an earlier version of the software.
Impact:
System boots to a halted state.
Workaround:
Before booting to the partition with the earlier version, delete /shared/bin/big3d.
Note: This issue might have ramifications for DNS/GTM support. DNS/GTM is not FIPS certified.
Component: Local Traffic Manager
Symptoms:
When using DTLS 1.0 the following two PFS ciphers are no longer negotiated and they cannot be used in a DTLS handshake/connection.
* ECDHE-RSA-AES128-CBC-SHA
* ECDHE-RSA-AES256-CBC-SHA
Conditions:
DTLS 1.0 is configured in an SSL profile.
Impact:
ECDHE-RSA-AES128-CBC-SHA and ECDHE-RSA-AES256-CBC-SHA are unavailable.
Component: Global Traffic Manager (DNS)
Symptoms:
The 'gtm_add' script fail to sync configuration information from the peer when 'options inet6' is present in /etc/resolv.conf.
Conditions:
The option 'options inet6' is used in /etc/resolv.conf.
Impact:
The 'gtm_add' script removes the current config and attempts to copy over the config from the remote GTM. When the remote copy fails, the local device is left without any config.
Workaround:
Remove the 'options inet6' from /etc/resolv.conf.
Component: TMOS
Symptoms:
Reconfiguring existing iapp results in an error:
01070712:3: Values (ICALL_SCRIPT_/Common/my_send_stats) specified for tcl user proc graph
(ICALL_SCRIPT_/Common/my_send_stats ICALL_SCRIPT_/Common/my_send_stats:escape_json): foreign key index
(name_FK) do not point at an item that exists in the database.
Conditions:
Reconfigure existing application while iCall is in use
Impact:
Modification of a running iCall script results in an error
Workaround:
None.
Component: TMOS
Symptoms:
The BIG-IP device silently reboots. After rebooting, you see the following in /var/log/messages:
info kernel: lasthop-ingress-v4: tcp non-syn packet
info kernel: syslog-ng[13461]: segfault at 0 ip 00002b17f748cc9c sp 00007fffedf7df78 error 4 in libc-2.17.so[2b17f740a000+1be000]
info kernel: syslog-ng[29749]: segfault at 2 ip 00002ba26ed77030 sp 00007ffc945207b8 error 4 in libglib-2.0.so.0.5000.3[2ba26ed3d000+10f000]
info kernel: audit_forwarder[31495]: segfault at 7a2da208 ip 0000000056e97be8 sp 00000000ffa9bc80 error 4 in libc-2.17.so[56e66000+1bf000]
Conditions:
Invalid syslog-ng server configuration or broken connection from BIG-IP toward configured syslog-ng remote server. Indication that this is occurring is the following entry from syslog-ng in /var/log/messages:
notice syslog-ng[3515]: Syslog connection broken; fd='54', server='AF_INET(10.220.79.129:514)', time_reopen='60'
Impact:
Syslog-ng crash/restart. Sometimes, this may lead to a silent reboot of BIG-IP also. Traffic disrupted while BIG-IP restarts.
Workaround:
Ensure syslog-ng server configuration is valid, and that the server is reachable.
Component: TMOS
Symptoms:
When using multiple virtual-servers sharing the same destination address (virtual-address), deleting a single virtual-server that contributes to a virtual-address status might not trigger a route-injection update.
Conditions:
-- Multiple virtual-servers sharing the same destination address.
-- Virtual-server is deleted.
Impact:
Route remains in a routing table and/or Route is not removed from a routing table.
Workaround:
Disable and re-enable the virtual-address after deleting a virtual-server.
Component: TMOS
Symptoms:
TMM may prematurely shut down (during its initialization) when several TMMs and interfaces are configured. The system logs messages in one or more TMM log files (/var/log/tmm*):
MCP connection aborted, exiting.
Conditions:
-- BIG-IP Virtual Edition (VE).
-- Happens during TMM startup.
-- The issue is intermittent, but is more likely to occur on systems with a lot of TMMs (more 20 in most cases) and several interfaces (approximately 8 or more).
Impact:
TMM shuts down prematurely. Traffic disrupted while tmm restarts. Possible deadlock and MCP restart loop requiring a full reboot of the BIG-IP device.
Workaround:
None.
Component: Service Provider
Symptoms:
MRF SIP in LoadBalancing Operation Mode inserts a VIA header to SIP request messages. This Via header is removed from the returned response message. The VIA header contains encrypted routing information to route the response message. The SIP specification states that INVITE/CANCEL messages in a dialogue should contain the same branch header. The code used to encrypt the branch field returns a different branch ID for INVITE and CANCEL messages.
Conditions:
-- Enabling SIP Via header insertion on the BIG-IP system.
-- SIP MRF profile.
-- Need to cancel an INVITE.
-- INVITE Via header received with multiple comma-separated values.
Impact:
Some SIP clients have code to verify the branch fields in the Via header. These clients expect the branch to be same for INVITE and CANCEL in a dialogue. Because the branch received is different, these clients are unable to identify the specific INVITE transaction. CANCEL is received and client sends a 481 error:
SIP/2.0 481 Call/Transaction Does Not Exist.
Workaround:
Use iRules to remove the topmost Via header and add new a new Via header that uses the same branch as INVITE and CANCEL while sending messages to SIP clients.
Component: TMOS
Symptoms:
Running the command 'tmsh list /sys db scheduler.minsleepduration.ltm'
shows that the value is -1.
The db variable 'scheduler.minsleepduration.ltm' is set to -1 on mcpd startup.
This overwrites a custom value.
Conditions:
-- The db variable 'scheduler.minsleepduration.ltm' has a non-default value set.
-- A reboot occurs.
Impact:
The db variable 'scheduler.minsleepduration.ltm' reverts to the default value. When the db variable reverts to the default value of unset -1, tmm uses more CPU cycles when idle.
Workaround:
Component: Local Traffic Manager
Symptoms:
BIG-IP is not compliant with a NIST revision to the SP800-56A standard for cryptographic algorithms.
Conditions:
Using cryptographic algorithms covered by this revision in a FIPS 140-2 deployment.
Impact:
BIG-IP will comply with the older standard.
Component: Local Traffic Manager
Symptoms:
When using the in-TMM monitoring feature, monitored targets (nodes/pool members) may be marked DOWN unexpectedly if there is a delay in responding to ping attempts.
Specifically, if the ping response from the target is delayed by more than the 'interval' value configured for the monitor, but less than the 'timeout' value configured for the monitor, the target may be marked DOWN.
Conditions:
This may occur when either:
-- In-TMM monitoring is enabled (sys db bigd.tmm = enable) and the monitor type uses in-TMM monitoring; OR
-- Bigd is configured to NOT reuse the same socket across consecutive ping attempts (sys db bigd.reusesocket = disable)
-- The monitored target does not respond to ping attempts within the 'interval' value configured for the monitor.
Impact:
The monitored target may be marked DOWN if it does not respond to ping attempts within the 'interval' value configured for the monitor, instead of within the 'timeout' value configured for the monitor.
Workaround:
To work around this issue, use one of the following methods:
-- Disable in-TMM monitoring and enable bigd socket reuse (sys db bigd.tmm = disable, and sys db bigd.reusesocket = enable).
-- Configure the monitor with an 'interval' value longer than the expected response time for the monitored target(s).
Component: TMOS
Symptoms:
-- Connections to self-IP addresses may fail.
-- SYNs packets arrive but are never directed to the Linux host.
Conditions:
-- Provision as vCMP dedicated host.
-- Create a self-IP address with appropriate allow-service.
Impact:
Repeated attempts to connect to the self-IP (e.g., via ssh) fail.
Workaround:
Component: Local Traffic Manager
Symptoms:
The iRule command HTTP::payload is not supported for use within Policy Enforcement Manager (PEM) policies. Attempting to use this within your configuration may result in the TMM crashing.
Conditions:
-- Policy Enforcement Manager (PEM) policy containing the iRule command HTTP::payload
Impact:
Traffic disrupted while the TMM restarts.
Workaround:
Do not use the iRule command HTTP::payload within Policy Enforcement Manager (PEM) policies.
Component: Advanced Firewall Manager
Symptoms:
The 'total port blocks' value, which can be found in PBA 'tmctl' tables, 'tmsh show', and SNMP, is limited to 64 bits of range. The upper 64 bits of the value are not taken into account.
Conditions:
This always occurs, but affects only systems whose configuration makes the 'total port blocks' value exceed 64 bits of range.
Impact:
Incorrect statistics.
Workaround:
None.
Note: For those who really need this value, it is still possible to manually calculate it, but that is not a true workaround.
Component: Local Traffic Manager
Symptoms:
Flow forwarding does not work with statemirror.verify enabled and source-port is preserve strict. Depending on the number of tmms and the IP addresses/ports on the network, this causes return traffic to get dropped.
Traffic captures show packets leaving the BIG-IP system on one tmm and being returned on another. The return traffic that encounters the second tmm is dropped.
Conditions:
-- Mirroring is enabled.
-- High availability (HA) peer is connected.
-- The source-port setting is preserve-strict.
-- The statemirror.verify option is enabled.
-- There is more than one tmm.
Impact:
Server-side return traffic to the BIG-IP is dropped. This causes connection timeouts and resets.
Workaround:
-- Disable statemirror.verify, disable source-port preserve-strict, disable mirroring.
-- On BIG-IP Virtual Edition (VE), add the following to tmm_init.tcl on both units and restart tmm:
ndal ignore_hw_dag yes
Component: TMOS
Symptoms:
The ip-tos-to-client setting is intended to apply to client traffic only. Server traffic IP ToS values are configured on the pool using the ip-tos-to-server property.
This is a change in behavior in that previously ip-tos-to-client was loosely interpreted to apply to server traffic as well if not overwritten by the pool.
Conditions:
The ip-tos-to-client value is set in the L4/L7 profile.
Impact:
IP ToS values in traffic to the server is unmodified.
Workaround:
Use the ip-tos-to-server property on pools to change IP ToS values to the server.
Alternatively, use an iRule to set the IP ToS to the server. For example:
when SERVER_CONNECTED {
IP::tos 63
Component: Application Security Manager
Symptoms:
The number '1' is added to the signature reference in violation details in the Request Log.
Conditions:
You click the '?' icon near signature name to view signature details and there are references for this signature
Impact:
The number 1 is shown before the link
Component: TMOS
Symptoms:
In the GUI, while trying to list a large configuration, tomcat restarts with error java.lang.OutOfMemoryError: Java heap space due to a large read operation.
Conditions:
-- From the GUI, navigate to Local Traffic :: Pools :: Pool List.
-- The configuration contains approximately 10,000 objects (objects include pools, nodes, virtual servers, etc.).
Impact:
When the system attempts to list the large configuration, tomcat restarts, resulting in 503 error.
Workaround:
Use the provision.tomcat.extramb database variable to increase the maximum amount of Java virtual memory available to the tomcat process.
Impact of workaround: Allocating additional memory to Apache Tomcat may impact the performance and stability of the BIG-IP system. You should perform this procedure only when directed by F5 Technical Support after considering the impact to Linux host memory resources.
-- Using a utility such as free or top, determine if you have enough free memory available to use this procedure.
For example, the following output from the free utility shows 686844 kilobytes available:
total used free shared buffers cachedMem:
16472868 15786024 686844 807340 827748 2543836
-/+ buffers/cache: 12414440 4058428
Swap: 1023996 0 1023996
-- View the current amount of memory allocated to the tomcat process by typing one of the following commands:
ps | grep " -client" | egrep -o Xmx'[0-9]{1,5}m'
The command output appears similar to the following example:
Xmx260m
Xmx260m
-- View the current value of the provision.tomcat.extramb database variable by typing the following command
tmsh list /sys db provision.tomcat.extramb
-- Set the provision.tomcat.extramb database variable to the desired amount of additional memory to be allocated using the following command syntax:
modify /sys db provision.tomcat.extramb value <MB>
-- If the device is part of a high availability (HA) configuration, the provision.tomcat.extramb database value should be synchronized to the peer devices from the command line. To run the ConfigSync process, use the following command syntax:
tmsh run /cm config-sync <sync_direction> <sync_group>
For example, the following command pushes the local device's configuration to remote devices in the Syncfailover device group:
tmsh run /cm config-sync to-group Syncfailover
-- Restart the tomcat process by typing the following command:
restart /sys service tomcat
Component: Application Security Manager
Symptoms:
An incorrect context (request) is logged for URL signatures in the violation details field.
Conditions:
-- ASM is running with a remote logger that has the violation_details field assigned.
-- A URL signature is matched.
Impact:
The logs do not provide the correct information, which might result in confusion or the inability to use the logged information as intended.
Workaround:
None.
Component: Application Security Manager
Symptoms:
Virtual Server is not created when a new HTTPS virtual server is defined with a new security policy using GUI page.
Conditions:
- In GUI open "create new policy" page
- Assign a policy name in "Policy Name"
- Assign "Fundamental" for "Policy Template"
- Choose "Configure new virtual server" in "Virtual Server"
- Select "HTTPS" for "What type of protocol does your application use?"
- Choose an IP address and port 443 for "HTTPS Virtual Server Destination"
- Choose an IP address and port 443 for "HTTPS Pool Member"
- "clientssl" for "SSL Profile (Client)"
- "serverssl" for "SSL Profile (Server)"
- click on "Save" button to create new policy
Impact:
The virtual server is not created when you create the policy, and HTTPS security will not be applied to this newly created security policy.
Workaround:
Go to the Local Traffic section and create a HTTPS Virtual Server manually. After that, assign the security policy to the Virtual Server.
Component: Global Traffic Manager (DNS)
Symptoms:
Newly added DNS/GTM device may sync old local config to other devices in the sync group.
Conditions:
Newly added DNS/GTM device has a more recent change than other devices in the sync group.
Impact:
Config on other DNS/GTM devices in the sync group are lost.
Workaround:
You can use either of the following workarounds:
-- Make a small DNS/GTM configuration change before adding new devices to the sync group.
-- Make a small DNS/GTM configuration change on the newly added device to re-sync the correct config to other DNS/GTM devices.
Component: Local Traffic Manager
Symptoms:
As the BIG-IP system attempts to open a TCP connection to a server-side object (e.g., a pool member), retransmissions of the initial SYN segment incorrectly use a non-zero acknowledgement number.
Conditions:
This issue occurs when the following conditions are true:
-- Standard TCP virtual server.
-- TCP profile with Verified Accept enabled.
-- Receipt of the client's ACK (as part of the client-side TCP 3-way handshake) is delayed. Due to Verified Accept being enabled, this delay causes the BIG-IP system to retransmit its SYN to the server until the client's ACK is received.
Impact:
Depending on the specific server implementation, or the security devices present on the BIG-IP system's server-side before the server, a SYN containing a non-zero acknowledgement number may be rejected. In turn, this may cause connections to fail to establish.
Workaround:
If compatible with your application and specific needs, you can work around this issue by disabling Verified Accept in the TCP profile.
Component: Local Traffic Manager
Symptoms:
The system kernel started spontaneously logging messages at an extremely high rate (~3000 per second):
Warning kernel: EMU(3)_INT: 0x0000000000000020
warning kernel: sum_sbe: 0
warning kernel: sum_dbe: 0
warning kernel: sum_wd: 0
warning kernel: sum_gi: 0
warning kernel: sum_ge: 0
warning kernel: sum_uc: 1
The above set of messages seems to be logged at about 2900-3000 times a second.
These messages continue after TMM fails its heartbeat and is killed. The system is rebooted by the host watchdog.
Conditions:
These messages are triggered by Nitrox5 driver when EMU microcode cache errors corrected by hardware.
Impact:
High rate of logging messages. The tmm heartbeat eventually fails, and tmm is restarted. Traffic disrupted while tmm restarts.
Workaround:
None.
Component: Global Traffic Manager (DNS)
Symptoms:
The DNS/GTM big3d daemon does not accept ECDH or DH ciphers.
Conditions:
Connections to big3d with ECDH or DH ciphers.
Impact:
ECDH/DH ciphers do not work with big3d.
Workaround:
Do not use ECDH/DH ciphers.
Component: Global Traffic Manager (DNS)
Symptoms:
Bind does not match IPv6 addresses configured for a zone view, and returns REFUSED responses, rather than the
expected answers.
After enabling debug logging in bind (see K14680), the apparent source address of the IPv6 DNS requests shows as being in the fe80::/96 range, rather than the IPv6 source address that sent the request.
For example:
debug 1: client @0x579bf188 fe80::201:23ff:fe45:6701%10#4299: no matching view in class 'IN'
Conditions:
- BIG-IP DNS is provsioned
- One or more ZoneRunner views is defined using IPv6 addresses.
- A DNS query is sent from an IPv6 source address
Impact:
You cannot use DNS views in bind (zonerunner) based on IPv6 addresses.
Workaround:
If possible, use only IPv4 addresses to define views for DNS queries
Component: TMOS
Symptoms:
SNMP request is times out.
Conditions:
This issue happens with TMOS version v15.1.0.4 or beyond after a IPv6 trap destination is configured.
Impact:
No response is returned for SNMP request.
Workaround:
Restart SNMP daemon by running the following TMSH command:
restart sys service snmpd
Component: TMOS
Symptoms:
Loading a UCS file with non-empty fipserr files can cause a FIPS-based system to remain offline.
Conditions:
-- Using a BIG-IP with a Platform FIPS license.
-- Loading a UCS file with a non-empty fipserr file.
Impact:
System is completely offline with spurious 'fipserr' failures, even after loading the UCS file.
Workaround:
Before creating a UCS archive, truncate the following files so they have zero size:
/config/f5_public/fipserr
/var/named/config/f5_public/fipserr
/var/dnscached/config/f5_public/fipserr
This can be accomplished using a command such as:
truncate -c -s0 /config/f5_public/fipserr /var/named/config/f5_public/fipserr /var/dnscached/config/f5_public/fipserr
Component: Global Traffic Manager (DNS)
Symptoms:
GUI reports an error when modifying DNS/GTM HTTP/HTTPS monitors:
01020036:3: The requested monitor parameter (/Common/http-default 2 RECV_STATUS_CODE=) was not found.
Conditions:
RECV_STATUS_CODE has never been set for the DNS/GTM HTTP/HTTPS monitors.
Impact:
Not able to make changes to DNS/GTM HTTP/HTTPS monitors through GUI.
Workaround:
If 'recv-status-code' has never been set, use tmsh instead.
Note: You can set 'recv-status-code' using tmsh, for example:
tmsh modify gtm monitor http http-default recv-status-code 200
Component: Advanced Firewall Manager
Symptoms:
When retrieving rule stats on a firewall policy, if two rules that share the same name but one of which is directly attached to the policy while the other is attached via a rule list, then a invalid JSON is returned. The JSON has identical keys for each entry associated with the rule. This is an invalid JSON structure that cannot be parsed correctly (Or data for one of the rules is lost)
Conditions:
A firewall policy that has one of rule directly attached to the policy while the other is attached via a rule list, and both rules share the same name.
Impact:
Invalid JSON structure returned for stat REST API call
Workaround:
Ensure that no rule shares its name with another rule.
Component: Local Traffic Manager
Symptoms:
When a virtual server with a SIP profile has mirroring enabled, connections on the standby unit will be removed shortly after being created. If tm.rstcause.log is enabled the reset cause message appears similar to the following:
-- err tmm1[18383]: 01230140:3: RST sent from 10.2.207.208:5080 to 10.2.207.201:31597, [0x2c1c820:1673] MBLB internal error (Routing problem).
-- err tmm1[18383]: 01230140:3: RST sent from 10.2.207.209:51051 to 10.2.207.202:5080, [0x2c1c820:931] MBLB internal error.
Conditions:
-- BIG-IP Virtual Edition (VE) running on 2000/4000 platforms using RSS hash.
-- Platforms with more than 1 tmm.
Impact:
Mirrored connections on the standby unit are removed.
Workaround:
-- On BIG-IP VE, add the following to tmm_init.tcl on both units and restart tmm:
ndal ignore_hw_dag yes
Component: Advanced Firewall Manager
Symptoms:
Version upgrade takes more time than usual when the config contains address-lists with a lot of IP addresses. The same delay will be observed with 'tmsh load sys config' as well.
Conditions:
-- Configure address-list with 10K to 20K IP addresses or address ranges or subnets.
-- Attempt upgrade / reboot of the platform.
Impact:
Version upgrade / 'tmsh load sys config' process takes a long time than usual.
Workaround:
1) Convert continuous individual addresses in the address-lists to IP address ranges and subnets if possible.
2) Remove the huge address-lists from config before the upgrade and add back after the upgrade process is finished.
Component: Application Security Manager
Symptoms:
Violation is missing/details not populated in the event log page, when a POST request with large number of parameters are sent to the BIG IP system.
Conditions:
-- A large POST request with lots of parameters is sent to BIG-IP system.
-- 'Learn New Parameters' is enabled.
Impact:
You cannot see the violation details.
Workaround:
Disabling parameter learning helps.
Note: This happens only with a large number of parameters. Usually it works as expected.
Component: Global Traffic Manager (DNS)
Symptoms:
GTM virtual servers have the wrong status (up when they should be down, or down when they should be up).
Conditions:
-- The GTM virtual servers are monitored with an HTTP or HTTPS monitor that performs HTTP status matching.
-- The status code (for example, 200) being searched for in the response appears elsewhere than in the first line (for example, in a following header).
Impact:
The system incorrectly matches the status code in a response line which is not the Status-Line. As a result, the availability status reported for a virtual server may be incorrect. This may cause the GTM system to send traffic to unsuitable resources causing application disruptions.
Workaround:
You can work around this issue by not performing HTTP status matching in your HTTP/HTTPS GTM monitors.
Component: Local Traffic Manager
Symptoms:
SAML Single log out (SLO) fails on BIG-IP platforms. The SAML module on the BIG-IP system reports following error messages:
-- SAML SSO: Error (12) Inflating SAML Single Logout Request
-- SAML SSO: Error (12) decoding SLO message
-- SAML SSO: Error (12) extracting SAML SLO message
Conditions:
Failures occur with Redirect SLO.
Impact:
SAML single logout does not work.
Workaround:
Use POST binding SLO requests.
Component: TMOS
Symptoms:
TMM crashes when the maximum transmission unit (MTU) or 'Use PMTU' setting is changed while passing IPsec traffic.
Conditions:
-- IPsec tunnel configured and passing traffic.
-- The MTU or 'Use PMTU' setting for the IPsec tunnel (in interface mode) is changed.
Impact:
Traffic disrupted while tmm restarts.
Workaround:
Do not change MTU or PMTU settings for the tunnel while it is passing traffic.
The settings can be changed while passing traffic, but TMM may crash very soon after the change. If the settings are changed and TMM does not crash soon after, then it will not spontaneously crash at some later point.
Component: TMOS
Symptoms:
The BIG-IP system does not log FIPS power-up self-test failures to the console.
Conditions:
A FIPS failure occurs during the power-up self test.
Impact:
Platform FIPS failures are made more difficult to identify and diagnose, because the system console fails to include anything at all that indicates a failure.
Workaround:
None.
Component: Application Visibility and Reporting
Symptoms:
TMM crashes with a core
Conditions:
AFM and AVR provisioned and collecting ACL statistics.
Impact:
Traffic disrupted while tmm restarts.
Workaround:
Disable the server-side statistics collection for the Network Firewall Rules using the following menu path:
Security :: Reporting : Settings : Reporting Settings : Network Firewall Rules.
Component: Local Traffic Manager
Symptoms:
Network flows are reset and errors are found in /var/log/ltm:
Route domain not reachable (strict mode).
Conditions:
Either:
-- LTM with iRules configured.
-- The iRule directs traffic to a node that is in a route domain.
-- LTM with an LTM policy configured.
-- The policy directs traffic to a node that is in a route domain.
Impact:
Traffic is not sent to the node that is in a route domain.
The iRule 'node' method and/or LTM policy 'node' specification require a route_domain to be specified in order for the traffic to be sent to a node that is assigned to a route domain.
Workaround:
Specify the node along with Route domain ID.
For iRules, change from this:
when HTTP_REQUEST {
node 10.10.10.10 80
To this (assuming route domain 1):
when HTTP_REQUEST {
node 10.10.10.10%1 80
For LTM policies, change from this:
actions {
forward
select
node 10.2.35.20
To this (assuming route domain 1):
actions {
forward
select
node 10.2.35.20%1
Component: TMOS
Symptoms:
The turboflex-low-latency and turboflex-dns profiles are no longer available in 15.1.x and 16.0.x software releases.
Conditions:
The turboflex-low-latency or turboflex-dns in use.
Impact:
Unable to configure turboflex-low-latency or turboflex-dns profiles after an upgrade to 15.1.x or 16.0.x software release.
Workaround:
None.
Component: Application Visibility and Reporting
Symptoms:
There are 2 issues:
1) tmsh analytics commands related to server side connections changed in 14.x
2)"max-tps" is non-cumulative and cannot be used in this context
Conditions:
Using tmsh analytics commands in BIG-IP v14.x and higher
Impact:
1) tmsh commands (related to server side connections) should be written differently in 14.x+
2)The "max-tps" measure is not applicable in the 'client-ip' context
Workaround:
There are 2 issues:
1)tmsh display name changed from total-server-side-conns to server-side-conns etc (14.x+)
2)Change the "max-tps" attribute "commutative" from false to true and change the merge formula from SUM to MAX
Component: Global Traffic Manager (DNS)
Symptoms:
Zonerunner GUI hangs when attempting to list Resource Records; mcpd times out.
Conditions:
Attempt to list Resource Records in Zonerunner GUI.
Impact:
Zonerunner hangs.
Workaround:
Zonerunner GUI is unusable until this issue is resolved. Use tmsh.
Component: TMOS
Symptoms:
The plane process statistics can be used to track the statistics of processes even though the process ID has changed over time. The processes are characterized as belonging to the control plane, data plane, or analysis plane. Some of the processes are incorrectly labeled.
Conditions:
Viewing the plane process statistics when diagnosing plane usage on the BIG-IP system.
Impact:
The percentage of usage of each plane can be confusing or incorrect.
Workaround:
None.
Component: Access Policy Manager
Symptoms:
For Access Policy created with Customization type: modern, Portal Access resource is still present on user's webtop after the checkbox "Publish on Webtop" is disabled in config
Conditions:
-- Access Policy created with Customization type: modern
-- Disable the checkbox "Publish on Webtop" for any Portal Access resource
Impact:
Disabled Portal Access resource visible on the webtop when it should be hidden.
Workaround:
Re-create Access Policy with Customization type: standard
Component: Application Security Manager
Symptoms:
BIG-IP issues a redirect to the content profile form that contains relevant violation details in the Request Log. If you follow this redirect and try to update profile, the action fails.
Conditions:
This occurs if you follow the redirect to the content profile page from the violation details page in the Request Log, and then try to update the profile
Impact:
You are unable to update the content profile.
Workaround:
Go to the list content profile page, and update the content profile from there.
Component: Local Traffic Manager
Symptoms:
When configured with a large number of in-TMM monitors, Nodes or Pool Members may not be marked DOWN immediately after the configured timeout period once the target stops responding to pings.
Conditions:
This may occur when:
-- In-TMM monitoring is enabled (via sys db bigd.tmm).
-- A large number of Nodes and/or Pool Members (several hundreds or thousands) are configured and monitored.
Impact:
Nodes or Pool Members which are not responsive may not be marked DOWN in a timely fashion.
Workaround:
You can work around this issue by disabling in-TMM monitoring, at the expense of decreased monitoring performance (higher CPU usage by the bigd daemon).
Component: Local Traffic Manager
Symptoms:
BIG-IP systems provide an option to sanitize HTTP traffic via the http_security profile. When the profile is configured to alarm on a violation, it is possible that a connection to the violating client is reset if a remote logging server is marked down.
Conditions:
-- A BIG-IP system has an HTTP profile and and an http_security profile with the alarm option set.
-- A remote logging server is configured via a BIG-IP pool.
-- The pool has a monitor that marks all the pool members down.
-- A request with an HTTP violation is processed and triggers an alarm configured in the http_security profile.
Impact:
-- A TCP connection to a client is reset by the BIG-IP system.
-- The web page may not render, or may not render as expected.
-- Data are not delivered to a server with a POST request.
Workaround:
None.
Component: TMOS
Symptoms:
Datasync-global-dg is in 'sync pending' status and is not automatically synced as expected.
Conditions:
Browser Challenges update image is automatically downloaded.
Impact:
Peers are not synced.
Workaround:
Manually sync datasync-global-db group.
Component: Local Traffic Manager
Symptoms:
If you overwrite the certificate that is configured on the server SSL profile and used with the HTTPS monitor, the BIG-IP system still uses an old certificate.
After you update the certificate, the stored certificate is incremented, but monitor logging indicates it is still using the old certificate.
Conditions:
--Create a pool with an HTTPS pool member.
--Create an HTTPS monitor with cert and key.
--Assign the HTTPS monitor to the HTTPS pool.
--Update the certificate via GUI or tmsh.
Impact:
The monitor still tries to use the old certificate, even after the update.
Workaround:
Use either of the following workarounds:
-- Restart bigd:
bigstart restart bigd
-- Modify the server SSL profile cert key, set it to 'none', and switch back to the original cert key name.
The bigd utility successfully loads the new certificate file.
Component: TMOS
Symptoms:
Loading an SCF configuration file does not restore file objects from the SCF's associated tar file.
Restoring the SCF fails with an error similar to this if the running configuration does not already contain the file:
01070712:3: Failed: name (/Common/test-crt) Cache path (/config/filestore/files_d/Common_d/certificate_d/:Common:test-crt) does not exist and there is no copy in trash-bin to restore from.
Unexpected Error: Loading configuration process failed.
Conditions:
Restore an SCF archive that references file objects, e.g.:
-- SSL certificates
-- SSL keys
-- iFiles
Impact:
Restoring SCF does not restore contents of file objects.
Workaround:
None.
Component: Access Policy Manager
Symptoms:
Local user db is not synced to the standby device when it comes online after being forced offline.
Conditions:
Valid high availability (HA) configuration.
- Make the standby device forced offline
- create a new local db user in the online device
- bring back the standby device online.
Impact:
The newly created user is not synced to the standby device unless localdbmgr is restarted on the standby.
Workaround:
Component: Local Traffic Manager
Symptoms:
Memory leak in umem_alloc_80 through creating/deleting an LTM node object.
Conditions:
-- Create a node object.
-- Delete the node object.
Impact:
This gradually causes tmm memory pressure, and eventually severe outcome is possible such as aggressive swiper and tmm restart. Traffic disrupted while tmm restarts.
Workaround:
Refrain continuous creation/deletion of LTM nodes.
Component: TMOS
Symptoms:
The vCMP host might have external trunk with internal MAC addresses. This is visible via 'tmsh show net fdb'.
Conditions:
-- vCMP is provisioned and has guests deployed on it.
-- vCMP host uses trunks.
-- Create VLANs using trunks and assign it to guests.
-- Guests need to be in high availability (HA) configuration.
Impact:
Traffic processing is disrupted.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
Sockets and log files are closed and re-opened twice instead of one time when the half-open TCP monitor pings successfully.
Conditions:
This occurs when the half-open monitor pings successfully.
Impact:
Minor performance impact.
Workaround:
None.
Component: Global Traffic Manager (DNS)
Symptoms:
RESOLV::lookup returns an empty string.
Conditions:
The name being looked up falls into one of these categories:
-- Forward DNS lookups in these zones:
- localhost
- onion
- test
- invalid
-- Reverse DNS lookups for:
- 127.0.0.0/8
- ::1
- 10.0.0.0/8
- 172.16.0.0/12
- 192.168.0.0/16
- 0.0.0.0/8
- 169.254.0.0/16
- 192.0.2.0/24
- 198.51.100.0/24
- 203.0.113.0/24
- 255.255.255.255/32
- 100.64.0.0/10
- fd00::/8
- fe80::/10
- 2001:db8::/32
- ::/64
Impact:
RESOLV::lookup fails.
Workaround:
Use a DNS Resolver ('net dns') and RESOLVER::name_lookup / DNSMSG:: instead of RESOLV::lookup:
1. Configure a local 'net dns' resolver, replacing '192.88.99.1' with the IP address of your DNS resolver:
tmsh create net dns-resolver resolver-for-irules answer-default-zones no forward-zones add { . { nameservers add { 192.88.99.1:53 } } }
2. Use an iRule procedure similar to this to perform PTR lookups for IPv4 addresses:
proc resolv_ptr_v4 { addr_v4 } {
# Convert $addr_v4 into its constituent bytes
set ret [scan $addr_v4 {%d.%d.%d.%d} a b c d]
if { $ret != 4 } {
return
# Perform a PTR lookup on the IP address $addr_v4, and return the first answer
set ret [RESOLVER::name_lookup "/Common/resolver-for-irules" "$d.$c.$b.$a.in-addr.arpa" PTR]
set ret [lindex [DNSMSG::section $ret answer] 0]
if { $ret eq "" } {
# log local0.warn "DNS PTR lookup for $addr_v4 failed."
return
# Last element in '1.1.1.10.in-addr.arpa. 600 IN PTR otters.example.com'
return [lindex $ret end]
-- In an iRule, instead of:
RESOLV::lookup @192.88.9.1 $ipv4_addr
call resolv_ptr_v4 $ipv4_addr
Component: Application Security Manager
Symptoms:
-- Valid users complain they are unable to connect, or they are required to do CAPTCHA frequently.
-- On the BIG-IP device, 'Device ID Deletion' violations are being raised.
Conditions:
-- Bot Defense Profile with 'Device ID' mode set to 'Generate After Access' is attached to the virtual server.
-- Some change in the browser or computer hardware occurs.
-- Surfing to multiple qualified pages at the same time.
Impact:
Valid requests might be mitigated.
Workaround:
Raise the threshold of 'Device ID Deletion' anomaly (recommended value according to the webserver, is approximately 4).
Component: TMOS
Symptoms:
Management route lost after reboot, leading to no access to BIG-IP via management address.
Conditions:
This occurs in the following scenario:
1) Deploy 2NIC BIG-IP Virtual Edition (VE) in GCP (e.g., by following the steps described in Deploying the BIG-IP VE in Google Cloud - 2 NIC: Existing Stack with BYOL Licensing :: https://github.com/F5Networks/f5-google-gdm-templates/tree/v3.0.3/supported/standalone/2nic/existing-stack/byol).
2) Once VE deployment completes, connect to the BIG-IP mgmt address and check the BIG-IP routing table. (There should be two default routes in it.) Enable SSH on an external self IP (to be able to connect to BIG-IP system after the reboot).
3) Reboot BIG-IP VE.
4) After reboot completes, SSH connect to the external self IP and check the BIG-IP routing table.
Impact:
The default route via mgmt interface no longer exists in the routing table. Inability to connect to BIG-IP VE via management address after reboot. No management access to BIG-IP.
Workaround:
Use either of the following workarounds:
-- Delete the route completely and reinstall the route.
-- Restart mcpd:
bigstart restart mcpd
Component: TMOS
Symptoms:
The following symptoms may be seen when the HSB is experiencing a large number of XLMAC errors and is unable to recover from the errors. After attempting XLMAC recovery fails, the current behavior is to failover to the peer unit and go-offline and down links.
This can be seen the TMM logs:
-- notice The number of the HSB XLMAC recovery operation 11 or fcs failover count 0 reached threshold 11 on bus: 3.
-- notice HA failover action is triggered due to XLMAC/FCS errors on HSB1 on bus 3.
-- notice HSBE2 1 disable XLMAC TX/RX at runtime.
-- notice HA failover action is cleared.
Followed by a failover event.
Conditions:
It is unknown under what conditions the XLMAC errors occur.
Impact:
The BIG-IP system fails over.
Workaround:
Modify the default high availability (HA) action for the switchboard-failsafe to reboot instead of go offline and down links.
Component: Local Traffic Manager
Symptoms:
If a QUIC packet is deemed lost, but an ACK for it is then received, the cwnd is halved despite there being no actual packet loss. Packet reordering can cause this situation to occur.
Conditions:
A QUIC packet is deemed lost, and an ACK for it is received before the ACK of its retransmission.
Impact:
Inefficient use of bandwidth in the presence of packet reordering.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
Whenever you create Oracle monitors, and add a member to the monitor, every time the OpenSSL libraries are loaded for a new connection, high CPU usage occurs.
Conditions:
-- Create an Oracle LTM monitor.
-- Add a pool member to the Oracle monitor created.
-- Platform FIPS is licensed.
Impact:
High CPU Usage due to the loading of libraries whenever new connection is created.
Workaround:
None.
Component: TMOS
Symptoms:
VLANs with a name that that start with "eth" will cause tmm to fail and restart.
Conditions:
Vlan name that starts with "eth"
Impact:
Since tmm fails to start, the BIG-IP cannot serve traffic.
Workaround:
Rename all vlans that start with "eth"
Component: Application Security Manager
Symptoms:
When configuring an IP address whitelist in Bot Defense Profile, using a default Route Domain, and sending a request with an X-Forwarded-For header the request might not be whitelisted.
Conditions:
-- Bot Defense Profile is attached to virtual server.
-- Bot Defense Profile has an IP address whitelist configured.
-- Using default Route Domain.
-- Sending a request with X-Forwarded-For header.
-- Might require heavy traffic.
Impact:
Request from a whitelisted IP address is blocked.
Workaround:
Whitelist the IP address using an iRule.
Component: Application Security Manager
Symptoms:
ASM sends a new cookie in every response, even though there is no change to the cookie name-plus-value.
Conditions:
-- ASM enabled.
-- Hostname is configured in the ASM policy.
-- The pool member sends different cookie values each time.
Impact:
ASM sends a Set-Cookie in every response, and it always sets the same cookie value.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
The use of OCSP responder may cause X509 certificate store instances to be leaked, eventually causing memory pressure.
Conditions:
OCSP responder configured.
Impact:
TMM ssl_compat memory usage grows over time, eventually causing memory pressure, and potentially a traffic outage due to TMM restart. Traffic disrupted while tmm restarts.
Workaround:
No workaround.
Component: Local Traffic Manager
Symptoms:
Use of OCSP responder may cause small amounts of SSL memory to be leaked, eventually leading to memory pressure.
Conditions:
OCSP responder configured.
Impact:
TMM SSL memory usage grows over time, eventually causing memory pressure, and potentially a traffic outage due to TMM restart. Traffic disrupted while tmm restarts.
Workaround:
No workaround.
Component: Local Traffic Manager
Symptoms:
Use of OCSP responder may cause SSL handshake instances to be leaked eventually leading to memory pressure.
Conditions:
OCSP responder configured.
Impact:
TMM ssl_hs memory usage grows over time, eventually causing memory pressure, and potentially a traffic outage due to TMM restart.
Workaround:
No workaround.
Component: Application Security Manager
Symptoms:
Live Update configuration is not updated.
Conditions:
-- The BIG-IP device being removed from the device group is also the last commit originator. (You might encounter this on AWS as a result of auto-scale.)
-- A new device is added to the device group.
-- Initial sync is pushed to the new device.
Impact:
Automatic signature updates (ASU) fail to sync.
Workaround:
Make a spurious change to Live Update from another device in the group and sync it to the group, for example:
1. Set the 'Installation of Automatically Downloaded Updates' to Scheduled and save.
2. Then return the setting to its previous state, and save again.
Component: TMOS
Symptoms:
When debug mode is enabled, racoon2 logs packet payloads during IKE negotiation. When multiple proposals are present in an IKE_SA_INIT packet, the logging of the proposal payloads is incorrect.
Conditions:
The initiator sends more than one proposal.
Impact:
Diagnosing connection issues is more difficult.
Workaround:
During debugging, ignore IKE_SA_INIT packet dump in the logs.
Component: Advanced Firewall Manager
Symptoms:
Linux host connections are unreliable, and you see warning messages in /var/log/kern.log:
warning kernel: : [182365.380925] nf_conntrack: table full, dropping packet.
Conditions:
This can occur during normal operation for configurations with a large number of monitors, for example, 15 KB active entries.
Impact:
Monitors are unstable/not working at all.
Workaround:
1. Modify /etc/modprobe.d/f5-platform-el7-conntrack-default.conf
increasing the hashsize value:
options nf_conntrack hashsize=262144
2. Save the file.
3. Reboot the system.
Component: Carrier-Grade NAT
Symptoms:
LSN64 with hairpinning configured can lead to a tmm core in rare circumstances.
Conditions:
- LSN64 virtual server.
- Hairpinning enabled.
- FLOW_INIT iRule.
- Full proxy config.
Impact:
Tmm cores. Traffic disrupted while tmm restarts.
Workaround:
Disable full proxy config of hairpinning.
Component: Local Traffic Manager
Symptoms:
HTTPS monitor state is down when server_ssl profile cipher string has the value 'TLSv1_2'.
-- configured cipherstring TLSv1_2/TLSv1_1 is rejected by OpenSSL.
Conditions:
-- Pool member is attached with HTTPS monitor.
-- Monitor is configured with an SSL profile.
-- The configured server_ssl profile has cipher string as DEFAULT:!TLSv1_2.
Impact:
Pool status is down.
Workaround:
-- Enable 'in-tmm' monitoring.
-- Use SSL options available in the server SSL profile to disable TLSv1_2 or TLSv1_1 instead of cipher string.
-- Use the same cipher string with cipher group / cipher rule that is attached to the SSL profile.
Component: TMOS
Symptoms:
Current implementation causes httpd down and makes the GUI inaccessible as soon as you import a new cert.
Conditions:
Upload new cert using Import-type 'Certificate' option.
Impact:
The GUI inaccessible as soon as you import a cert using import-type 'Certificate'.
Workaround:
Manually copy the correct key.
Component: TMOS
Symptoms:
When installing an Engineering Hotfix, the following error may be logged in /var/log/liveinstall.log:
Argument "" isn't numeric in numeric eq (==) at /var/tmp/install/pkgcpio/usr/local/lib/tm_install/Hotfix.pm line 651.
Conditions:
This error may occur when installing an Engineering Hotfix, if the Engineering Hotfix does not include an update to the nash-initrd component.
Impact:
The error message gives a mistaken impression that the Engineering Hotfix did not install successfully. However, it does install correctly, and the system operates without issue. You can safely ignore this message.
Workaround:
None.
Component: Advanced Firewall Manager
Symptoms:
Syn-cookie protection of FastL4/TCP profile might get enabled when performing multiple software upgrades.
Conditions:
-- Performing an upgrade from version earlier than 14.0.0 to a version higher than or equal to 14.0.0.
-- Then performing another upgrade.
Impact:
Value of profile attribute syn-cookie-enable might be changed during an upgrade.
Workaround:
Save configuration before starting each upgrade.
Component: TMOS
Symptoms:
The Bidirectional Forwarding Detection (BFD) session for an IPv6 static route is not shown in response to the command:
imish -e "show bfd session"
Conditions:
-- BFD is configured with static route IPv6.
-- Restart the oamd process.
Impact:
BFD session is not shown in 'show bfd session'.
Workaround:
Restart tmrouted:
bigstart restart tmrouted
Component: Local Traffic Manager
Symptoms:
-- When 'clsh reboot' is executed on the primary blade, it internally calls ssh reboot on all secondary blades and then reboots the primary blade. The 'clsh reboot' script hangs, and there is a delay in rebooting the primary blade.
-- Running 'ssh reboot' on secondary blades hangs due to sshd sessions getting killed after network interface down.
Conditions:
-- Running 'clsh reboot' on the primary blade.
-- Running 'ssh reboot' on secondary blades.
Impact:
Secondary blade IPSec Security Associations (SAs) are lost, resulting in SA sync issues.
Workaround:
Perform a reboot from the GUI.
Component: Local Traffic Manager
Symptoms:
Log messages written to /var/log/ltm:
-- notice tmm2[30394]: 01010259:5: External Datagroup (/Common/dg1) queued for update.
-- notice panic: ../kern/sys.c:1081: Assertion "entry != NULL" failed.
Conditions:
-- Create datagroups.
-- Some condition causes a datagroup to not be present (e.g., delete, rename operations, or another, internal operation).
-- Load the config.
Impact:
Internal mapping of external datagroup fails. Datagroup creation fails.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
HTTP/3 aborts the connection upon receipt of a 'second' SETTINGS frame.
Conditions:
HTTP/3 first receives an incomplete SETTINGS frame, followed by more SETTINGS frame bytes.
Impact:
Connection fails to complete.
Workaround:
None.
Component: TMOS
Symptoms:
While deploying telemetry, restnoded uses excessive memory and restarts, and the telemetry consumer server is down.
Conditions:
Telemetry is deployed according to https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/prereqs.html
Impact:
Restnoded restarts and is not available
Component: TMOS
Symptoms:
When ASM is attached to a Virtual Server, a BIG-IP user account configured with the Guest role cannot see virtual server details. An error message is printed instead:
01070823:3: Read Access Denied: user (guestuser) type (GTM virtual score).
Conditions:
-- ASM Policy attached to virtual server.
-- Logging onto the BIG-IP system using an account configured with the guest user role.
-- Running the command:
tmsh show ltm virtual detail
Impact:
Cannot view virtual server details.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
HTTP/3 streams abort at seemingly arbitrary times when BIG-IP is receiving large amounts of data.
Conditions:
HTTP/3 receives an incomplete frame header on a given stream.
Impact:
Data transfer is incomplete.
Workaround:
Component: Access Policy Manager
Symptoms:
When the access profile type is OAuth-RS or ALL, and sends a request with a Bearer token longer than 4080 bytes in the Authorization header to the virtual server, OAuth fails with ERR_NOT_SUPPORTED.
Conditions:
Bearer token longer than 4080 bytes
Impact:
APM oauth fails with ERR_NOT_SUPPORTED.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
Earlier versions contained fixes that enforce several RFC compliance items for HTTP requests and response processing by BIG-IP systems. Enforcement for some of these items is unavoidable, but might cause issues for certain applications.
Conditions:
The configuration has a virtual server with an HTTP profile.
Impact:
Some applications that allows certain constructions after a header name may not function.
Workaround:
None.
Component: Application Security Manager
Symptoms:
Upon upgrade, active ASM policies are preserved, and inactive policies are deleted.
Conditions:
-- Configuration contains active and inactive ASM policies.
-- Upgrade the BIG-IP system to any later version.
-- You can check existing ASM policies in tmsh:
tmsh list asm policy
Impact:
Only the active ASM policies are preserved; the inactive policies are deleted.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
ICMP traffic to a disabled virtual-address might be handled by a virtual-server.
Conditions:
Virtual-server with an address space overlapping with a self-IP, capable of handling ICMP traffic, for example:
ip-forward wildcard 0.0.0.0/0 virtual-server
Impact:
ICMP traffic to a virtual-address might be handled by a virtual-server.
Workaround:
There is no workaround.
Component: Advanced Firewall Manager
Symptoms:
With some configurations, executing a command such as 'tmsh show security firewall global-rules active' loops continuously, causing stat counters to rise, and possibly log messages to be written to /var/log/ltm.
Conditions:
-- AFM is routing traffic to a Virtual Server through the 'Send to Virtual' option.
-- The target Virtual Server uses the 'LB_FAILED' iRule to select a new Virtual Server through virtual command and 'LB::reselect'.
Impact:
The iRule loops continuously, causing stat counters to rise, and possibly logging messages in /var/log/ltm.
Workaround:
This configuration should be avoided, but if it is used, and if this does happen, you can restart tmm:
bigstart restart tmm
This stops the current looping, until it is triggered again.
Impact of workaround: Traffic disrupted while tmm restarts.
Component: Local Traffic Manager
Symptoms:
HTTP/2 Clone pools are not working when the Clone Pool (Server) option is selected. This issue occurs when a HTTP/2 profile (Server) or HTTP/2 full-proxy configuration is enabled and an HTTP/2 clone pool is set on a virtual server. This issue prevents traffic from being copied to the appropriate clone pool member.
Conditions:
A virtual server provisioned with the following configuration:
--HTTP/2 default pool.
--HTTP/2 clone pool (server).
--HTTP/2 profile (server) or HTTP/2 profile full-proxy configuration.
Impact:
Clone pools (server) do not mirror HTTP/2 traffic.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
Node address field is disabled when trying to create custom HTTP monitor, so you cannot enter a node address This prevents you from using the Test operation to test this type of monitor in the GUI.
Conditions:
-- In a new monitor derived from HTTP, click the Test tab.
-- View the Address field, and then try to run the test.
Impact:
The Address field is disabled, with *.* in the field. You cannot enter a node address. The test fails with a message:
invalid monitor destination of *.*:80.
Workaround:
Use tmsh instead of the GUI.
Component: TMOS
Symptoms:
There there are thousands of FQDN nodes and thousands of pools that have FQDN pool members, mcpd can run out of memory during a full config sync.
The mcpd process might fail and restart or it might remain running but have its virtual memory so fragmented that queries to mcpd might fail to allocate memory.
One of signs that this has occurred is a non-zero free_fail count in the tmstat table vmem_kstat.
Conditions:
-- Thousands of FQDN nodes
-- Thousands of pools with FQDN pool members
-- Full config sync.
Impact:
-- The mcpd process might restart.
-- The config save operation fails:
tmsh save /sys config fails
-- Other queries to mcpd fail.
Workaround:
None.
Component: Access Policy Manager
Symptoms:
A SIGSEGV error occurs after a connection is ended. This is an intermittent issue that inconsistently recurs.
Conditions:
APM Per-Request is processing a flow that has already been reset (RST) by another filter, such as HTTP or HTTP/2.
Impact:
Connections might reset. You might experience a tmm crash. This is an intermittent issue.
Workaround:
None.
Component: Application Visibility and Reporting
Symptoms:
The virtual server might stay attached to the initial HTTP profile.
Conditions:
Attaching new HTTP profiles or just detaching an existing one.
Impact:
The virtual server stays attached to the former HTTP profile, meaning that the virtual server might be attached to a different HTTP profile than what the GUI displays. Configuration changes to the HTTP profile the GUI shows as attached are not reflected in the virtual server. For example, the new HTTP profile might enable XFF, but if the former attached profile does not enable it, the virtual server does not accept XFF.
Workaround:
Create new similar virtual server and attach it to the correct HTTP profile.
Component: Access Policy Manager
Symptoms:
If the Virtual Desktop Interface (VDI) plugin aborts, the names of the events are not logged in the APM log file.
Conditions:
- Virtual Desktop Interface (VDI) configured
- The plugin encounters a problem and aborts
Impact:
Event names are not displayed in the APM log.
Workaround:
None.
Component: Access Policy Manager
Symptoms:
Data plane performance for VDI use cases (Citrix/VMware proxy) is degraded during frequent access session statistic updates.
Conditions:
APM is used as VDI proxy for Citrix or VMware.
Impact:
APM's VDI proxy does not perform to its full capacity.
Workaround:
None.
Component: Policy Enforcement Manager
Symptoms:
When a PEM profile is associated with a protocol that can create dynamic server-side listeners (such as FTP), and source-address-translation is also enabled on the virtual server, traffic on that flow (for example ftp-data) is not associated with the subscriber, and is therefore not counted or categorized.
Conditions:
-- Listener configured with PEM and FTP profiles
-- Some form of source address translation is enabled on the listener (for example, SNAT, Automap, SNAT Pool)
Impact:
Inaccurate subscriber traffic reporting and classification.
Workaround:
None.
Component: Access Policy Manager
Symptoms:
OneConnect is a feature that reuses server-side connections. When WEBSSO is enabled, it always creates a new server-side connection, and does not reuse pooled connections.
Conditions:
Virtual server configured with both a WEBSSO and a OneConnect profile.
Impact:
Idle server-side connections that should be eligible for reuse by the virtual server are not used. This might lead to buildup of idle server-side connections, and may result in unexpected 'Inet port exhaustion' errors.
Workaround:
None.
Component: TMOS
Symptoms:
A device reboot occurs when pressing Ctrl-Alt-Del.
Conditions:
This occurs when pressing Ctrl-Alt-Del or sending the command to a BIG-IP Virtual Edition (VE) virtual console.
Impact:
Accidental reboots are possible. You should not reboot VE using Ctrl-Alt-Del.
Workaround:
Run the following command:
systemctl mask ctrl-alt-del.target
Component: Application Security Manager
Symptoms:
Fields in 'Logout Page' for a non-UTF8 security policy has incorrect encoding for values, including non-English characters in the GUI and iControl REST.
Conditions:
This can be encountered while creating a non-UTF8 security policy via iControl REST, where the 'expected' and 'unexpected' fields contain non-UTF8 content.
Impact:
Logout Page field values are displayed with the wrong encoding.
Workaround:
None.
Component: Application Security Manager
Symptoms:
Not all the CPUs are utilized. The CPUs that are not loaded are those with ID greater than 31.
Conditions:
BIG-IP is installed on a device with more than 32 cores.
Impact:
ASM does not use all of the available CPU cores.
Workaround:
1. Modify the following file on the BIG-IP system:
/usr/local/share/perl5/F5/ProcessHandler.pm
Change this:
ALL_CPUS_AFFINITY => '0xFFFFFFFF'
To this:
ALL_CPUS_AFFINITY => '0xFFFFFFFFFFFF',
2. Restart the asm process:
bigstart restart asm.
Component: TMOS
Symptoms:
BIG-IP Virtual Edition (VE) does not pass traffic when deployed on ESXi 6.5 hypervisors, when the VE is using VMXNET 3 network interfaces (VMXNET 3 interfaces are the default).
'tmsh show net interface' indicates that one or more interfaces are not initialized.
Conditions:
-- BIG-IP VE running on VMware ESXi 6.5 hypervisor.
Impact:
Traffic does not pass through non-mgmt interfaces.
Workaround:
-- On the BIG-IP systems, you can switch to the 'sock' driver.
Note: The workarounds that switch driver must be applied individually to devices, as they do not synchronize via ConfigSync.
IMPORTANT: The driver must be configured the same way on all devices in a sync-failover device group.
1, At the command prompt, run the following command to enable the sock driver in tmm_init.tcl:
echo "device driver vendor_dev 15ad:07b0 sock" >> tmm_init.tcl
2. Restart tmm:
tmsh restart sys service tmm
-- After restarting, the sock driver should be listed in the 'driver_in_use' column when running the following command.
tmctl -d blade tmm/device_probed
pci_bdf pseudo_name type available_drivers driver_in_use
------------ ----------- --------- --------------------- -------------
0000:03:00.0 F5DEV_PCI xnet, vmxnet3, sock,
0000:0b:00.0 1.1 F5DEV_PCI xnet, vmxnet3, sock, sock
0000:13:00.0 1.2 F5DEV_PCI xnet, vmxnet3, sock, sock
0000:1b:00.0 1.3 F5DEV_PCI xnet, vmxnet3, sock, sock
Component: Application Security Manager
Symptoms:
A scheduled live update does not occur at the scheduled time.
Conditions:
A scheduled installation is set for only a single day, between 00:00-00:14.
Impact:
Automated installation does not initiate
Workaround:
There are two options:
1. Install the update manually.
2. Set two consecutive days where the second day is the day with the schedule set between 00:00-00:14
Component: Anomaly Detection Services
Symptoms:
The watchdog process in the BIG-IP ASM monitors terminates the admd process.
Conditions:
On some heavy load configuration process, such as version upgrade.
Impact:
Restart of admd daemon. The restarts may be continuous. No stress-based anomaly detection or behavioral statistics aggregation until admd restarts.
Workaround:
For the case of continuous restarts, a partial solution is to disable admd during busy periods such as upgrades. To do so, issue the following two commands, in sequence, after the upgrade is complete:
bigstart stop admd
bigstart start admd
Component: Local Traffic Manager
Symptoms:
iRule commands issued after a clientside or serverside command operate on the wrong peer flow.
Conditions:
An iRule contains a script that parks in a clientside or serverside command.
Examples of parking commands include 'table' and 'persist'.
Impact:
The iRule commands operate on the wrong peer flow.
Workaround:
Avoid using commands that park inside the clientside or serverside command.
Component: TMOS
Symptoms:
Traffic that should be encapsulated and sent via tunnel might get dropped with an ICMP error, destination unreachable, unreachable route. This happens in a scenario where no route exists towards the remote tunnel endpoint and the BIG-IP system relies on autolasthop to send the encapsulated traffic back to the other end of the tunnel.
Conditions:
No route exists to the other end of the tunnel.
Impact:
Traffic dropped with ICMP error, destination unreachable, unreachable route.
Workaround:
Create a route towards the other remote end of the tunnel.
Component: Anomaly Detection Services
Symptoms:
False DoS attack detected. Behavioral DoS (ASM) might block legitimate traffic.
Conditions:
This can occur for some requests that have high latency and low TPS.
Impact:
False DoS attack detected. Behavioral DoS (ASM) can block legitimate traffic.
Workaround:
Modify the default sensitivity value from 50 to 500:
tmsh modify sys db adm.health.sensitivity value 500
For some sites with server latency issues, you might also have to increase the health.sensitivity value; 1000 is a reasonable number.
The results is that the attack is declared later than for the default value, but it is declared and the site is protected.
Component: Local Traffic Manager
Symptoms:
OneConnect allows load balancing of HTTP requests from the same client connection over a pool of server side connections. When NTLM authentication is used, the NTLM Conn Pool allows reuse of server-side connections for authenticated client side connections. It holds HTTP authentication headers which is no longer necessary once a client is authenticated.
Conditions:
-- The virtual server is configured with both OneConnect and NTLM Conn Pool profiles.
-- A large number of client systems with NTLM authentication are load balanced via the virtual server with long-lived connections.
Impact:
The BIG-IP system experiences memory pressure, which may result in an out-of-memory condition and a process crash, and potentially cause failover and interruption of traffic processing.
Workaround:
None.
Component: TMOS
Symptoms:
TMM may not start when using more than 11 network interfaces with more than 11 vCPUs configured.
In /var/log/tmm it can be seen that tmm0 is waiting for another TMM, e.g.:
-- notice ixlv(1.1)[0:5.0]: Waiting for tmm10 to reach state 1...
In the TMM log for that TMM, it can be seen that it is waiting for tmm0, e.g.:
-- notice ixlv(1.10)[0:6.0]: Waiting for tmm0 to reach state 2...
Conditions:
-- BIG-IP Virtual Edition (VE).
-- More than 11 interfaces configured.
-- More than 11 vCPUs configured.
Impact:
TMM does not start.
Workaround:
Configure fewer network interfaces or vCPUs.
Component: Application Security Manager
Symptoms:
BIG-IP systems send unexpected WebSocket server messages to the remote logging server.
Conditions:
-- ASM provisioned.
-- ASM policy and WebSocket profile attached to a virtual server.
-- More than one remote logging profile is attached to a virtual server.
-- One of the remote loggers has response-logging=all.
Impact:
Remote logging server overloaded with unexpected WebSocket messages.
Workaround:
Set response-logging=illegal in all remote logging profiles.
Component: TMOS
Symptoms:
Admin users are unable to log in.
Conditions:
-- Remote ldap auth enabled
-- Administrative users are authenticated with the "cert-ldap" source.
-- The admin user tries to log in
Impact:
The cert-ldap authentication does not work.
Workaround:
Manually edit the /etc/nslcd.conf and set the referrals to no.
Component: TMOS
Symptoms:
The tcpdump command exits immediately with an error:
errbuf ERROR:TMM side closing: Aborted
tcpdump: pcap_loop: TMM side closing: Aborted
Conditions:
Capturing the packets on tmm interfaces.
Impact:
Unable to capture the packets on specific tmm interfaces.
Workaround:
There are two possible workarounds:
-- Start tcpdump on the tmm that actually owns the interface using the TCPDUMP_ADDR command; for example, using blade1 for 1/0.16, run the command:
TCPDUMP_ADDR=127.1.1.16 tcpdump -w testcap.pcap -s 0 -i 1/0.16
-- Send the TCPDUMP_ADDR command to a specific tmm, which could work from any blade (127.20.<slot>.<tmmnumber+1> (e.g. 127.20.1.1 == slot1/tmm0, 127.20.2.16 == slot2/tmm15):
TCPDUMP_ADDR=127.20.1.16 tcpdump -w testcap.pcap -s 0 -i 1/0.16
Component: TMOS
Symptoms:
Attempting to configure and deploy an SSL Orchestrator configuration may result in a configuration processor timeout error.
Conditions:
When the TMOS control plane is taking a relatively high amount of CPU processing due to intense REST framework management workflows, any attempt to configure and deploy an SSL Orchestrator configuration using the guided configuration user interface, might take longer than the default 30 seconds timeout.
Impact:
The operation fails, resulting in a configuration processor timeout error. The BIG-IP administrator cannot configure and deploy an SSL Orchestrator configuration.
Workaround:
1. Edit the /var/config/rest/iapps/f5-iappslx-ssl-orchestrator/nodejs/gc/orchestratorConfigProcessor.js file:
vi /var/config/rest/iapps/f5-iappslx-ssl-orchestrator/nodejs/gc/orchestratorConfigProcessor.js
2. Locate the var _createComponentBlock = function(...) and locate the following line inside the function body (line ~1290):
requestBody.state = "BINDING";
3. Add the following after the 'requestBody.state' line:
requestBody.configProcessorTimeoutSeconds = 120;
4. Save the edited file.
5. Restart the restnoded daemon:
bigstart restart restnoded
Component: Local Traffic Manager
Symptoms:
When the BIG-IP system is configured to use the RAM Cache feature, a corresponding profile may report an excessively large value for the cache_misses_all counter under certain conditions.
Conditions:
-- The BIG-IP system has a virtual server with web-acceleration profile without an application (RAM Cache feature).
-- The virtual receives uncacheable requests which are interrupted by a client or are not served by a server.
Impact:
A value for cache_misses_all incurs an arithmetic overflow, and shows an excessive number comparable with 1.8e19. The issue has no functional impact; the system operates as normal.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
-- Increased baseline CPU.
- The memory_usage_stats table shows a continuous increase in mds_* rows.
Conditions:
Configure IPFIX log destination and make regular changes to the associated configuration.
Impact:
Increased baseline CPU may result in exhaustion of CPU resources.
Workaround:
Limiting changes to associated configuration can slow the effects of this issue.
Component: Local Traffic Manager
Symptoms:
BIG-IP is not compliant with a NIST revision to the SP800-56A standard for cryptographic algorithms.
Conditions:
Using cryptographic algorithms covered by this revision in a FIPS 140-2 deployment.
Impact:
BIG-IP will comply with the older standard.
Workaround:
Updated cryptographic key assurances and pair-wise consistency checks according to the SP800-56Arev3 standard.
Component: Application Security Manager
Symptoms:
When deleting (via tmsh) bot-related ordered list items like bot white-lists, bot-microservices, and bot-microservices URLs, an error occurs when adding and saving new items via GUI:
Bot defense profile <profile full name> error: match-order should be unique.
Conditions:
1.Create three items with consecutive match-orders values via tmsh, for example: three bot whitelist items, the first with match-order 1, the second with match-order 2, and the third with match-order 3.
2. Delete item with the value: match-order 2 (in tmsh), and save.
3. Switch to the GUI, add new whitelist item, and save.
Impact:
The system reports an error, and the bot configuration cannot be saved via GUI. However, dragging between items (and then dragging back) overcomes this error.
Workaround:
Drag between two items, and then drag back.
Component: Global Traffic Manager (DNS)
Symptoms:
When GTM/DNS systems in the same sync group receive the error 'SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca', these systems cannot automatically connect to BIG-IP devices with which that GTM/DNS device has not already exchanged a SSL cert.
As part of normal functionality, when one GTM/DNS tries to connect to a BIG-IP server and receives 'unknown ca' SSL error, if its peer GTM/DNS has already built a connection with that BIG-IP server, then the second GTM/DNS system should also be able to connect to that BIG-IP server automatically. But it cannot because of this issue.
The problem exists only when the GTM/DNS device has not exchanged a cert with the BIG-IP server object, and there are two or more certs in /config/httpd/conf/ssl.crt/server.crt on that GTM/DNS device.
You might see messages similar to the following:
-- iqmgmt_ssl_connect: SSL error:14094418:SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca.
-- err gtmd[28112]: 011ae0fa:3: iqmgmt_ssl_connect: SSL error: error:14094418:SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca (336151576).
-- notice gtmd[28112]: 011ae03d:5: Probe to 10.10.0.3: buffer = <direct><![CDATA[<clientcert><ip>10.10.0.10</ip><target_ip>10.10.0.6</target_ip><cert>....
Conditions:
-- /config/httpd/conf/ssl.crt/server.crt file with two or more certs on the requesting GTM/DNS device.
-- Configuration is as follows:
1. GTMDNS1 and GTMDNS2 are in a same GTM/DNS sync group.
2. GTMDNS1 has a self-authorized CA cert.
3. You add a BIG-IP server that is is reachable but with which GTMDNS1 has not exchanged SSL certs.
Impact:
Certain GTM/DNS systems in the sync group cannot automatically connect to BIG-IP devices as expected. You must run additional bigip_add commands on those GTM/DNS systems in the GTM/DNS sync group to add the BIG-IP server.
Workaround:
Run bigip_add on each GTM/DNS server to add the configured BIG-IP servers.
Component: Local Traffic Manager
Symptoms:
The HTTP session initiated by curl hangs.
Conditions:
-- The problem occurs when the file to be compressed meets the following criteria:
-- The following platforms with Intel QAT are affected:
+ B4450N (A114)
+ i4000 (C115)
+ i10000 (C116/C127)
+ i7000 (C118)
+ i5000 (C119)
+ i11000 (C123)
+ i11000 (C124)
+ i15000 (D116)
-- File size to be compressed is less than compression.qat.dispatchsize.
-- File size to be compressed is one of specific numbers from this list: 65535, 32768, 16384, 8192, 4096.
Impact:
Connection hangs, times out, and resets.
Workaround:
Use software compression.
Component: Local Traffic Manager
Symptoms:
With the HTTP RFC enforcement profile option enabled, incoming health monitor requests without an HTTP version in the request line (HTTP/0.9) may fail to produce the correct result for dual BIG-IP configurations. This can result in incoming health monitor traffic being incorrectly blocked when traveling through a virtual server. These health monitors will be unable to provide the correct availability of the intended resource. The default HTTP and HTTPS monitors as well as any custom monitors with a missing HTTP version in their requests could see this issue.
Conditions:
A dual BIG-IP configuration may provisioned with the following considerations.
-- One or more BIG-IP systems are in the network path for monitor traffic.
-- The first BIG-IP system uses a health monitor that initiates a health check request (without an HTTP version) in the request line (HTTP/0.9) against a second BIG-IP system.
-- The second (downstream) BIG-IP system has a virtual server that is the endpoint for the monitor. The virtual server is configured with an HTTP profile with the HTTP RFC Compliance profile option selected.
Impact:
Rather than receiving the correct health check result, the original BIG-IP system can fail to report whether the second BIG-IP is available.
Workaround:
You can use http_head_f5 monitor to perform health checks.
Component: Service Provider
Symptoms:
-- 'DIAMETER::host origin' command may not be set correctly.
There are errors in ltm/log:
err tmm[18562]: 014c0001:3: DIAMETER: hud_diam_handle error: Not found
Conditions:
-- Virtual server is configured with a diameter profile enabled with an ingress iRule, for example:
ltm rule Diameter - iRule {
when MR_INGRESS {
DIAMETER:: host origin "hostname.realm.example"
-- Traffic arrives containing CER and ULR messages.
Impact:
Using the iRule to change the host origin corrupts the diameter message.
Workaround:
None.
Component: TMOS
Symptoms:
All Bandwidth Controller (BWC) stats are 0 (zero) even though traffic is passing.
Conditions:
-- A BWC policy is attached to a virtual server.
-- The virtual server with the attached BWC policy is modified.
Impact:
The system disassociates the BWC policy from the virtual server. Traffic is no longer throttled according to the policy rules.
Workaround:
To reattach the policy, detach the Bandwidth Controller policy from the virtual server, and then reapply it.
Component: TMOS
Symptoms:
Tmm crashes while passing traffic through PEM.
Conditions:
-- PEM policy with bandwidth controller.
-- iRule makes a traffic decision based on certain unique PEM sessions.
Impact:
Traffic disrupted while tmm restarts.
Workaround:
None.
Component: TMOS
Symptoms:
After failover, the active BIG-IP system fails to respond to DPD requests from some of its eNB neighbors, which results in deletion of IKE tunnel peer as well as the BIG-IP system.
Conditions:
-- The BIG-IP is configured with more than 300 IKE/IPsec tunnels.
-- The BIG-IP system fails over.
Impact:
Since BIG-IP systems do not respond to DPD requests, eNB deletes the IKE tunnel after a few retries.
Workaround:
None.
Component: TMOS
Symptoms:
Repeatedly forcing VIPRION blade failover can cause the GUI to get stuck displaying:
This device is not operational because the loaded configuration contained errors or unlicensed objects. Please adjust the configuration and/or the license, and re-license the device.
Examination of the configuration and license shows that there are no configuration errors or unlicensed configuration objects. Failover occurs as expected, and the device is operational.
Conditions:
Repeatedly forcing blade failover triggers this issue, with commands such as the following issued in rapid succession:
- tmsh modify sys cluster default members { 2 { disabled } }
- tmsh modify sys cluster default members { 2 { enabled } }
- tmsh modify sys cluster default members { 1 { disabled } }
- tmsh modify sys cluster default members { 1 { enabled } }
Impact:
The GUI continuously displays the banner the device-not-operational banner. However, the device is operational.
Workaround:
You can use either of the following workarounds to return the GUI to normal operation:
-- Restart mcpd by running the following command:
bigstart restart mcpd
-- Reboot the VIPRION system.
Component: Global Traffic Manager (DNS)
Symptoms:
DNS Zone syncing is missing Resource Records.
Conditions:
This occurs when a large number of configuration changes, including Wide IP changes, are made simultaneously on multiple GTM/DNS devices in a sync group.
Impact:
DNS Resource Records can be missing from the BIND DNS database.
The impact of this is that if GSLB Load Balancing falls back to BIND, the DNS Resource Records may not be present.
Workaround:
Restrict configuration (Wide IP) changes to one GTM/DNS device in a device group.
Note: It is also possible to turn off Zone Syncing. GTM/DNS configuration is still synced, but the you lose the ability to sync non-Wide IP changes to the BIND DB. If you do not utilize ZoneRunner to add additional non-Wide IP records, this is a problem only when GSLB resorts to Fallback to BIND. This can be mitigated with DNSX and DNS (off device) for non Wide IP Resource Records.
Component: Local Traffic Manager
Symptoms:
UDP commands in iRules executed during FLOW_INIT event fail.
Conditions:
An iRule that contains UDP commands is executed on the FLOW_INIT event.
Impact:
UDP commands in iRules executed during FLOW_INIT event fail.
Workaround:
None.
Component: TMOS
Symptoms:
In the GUI if you change a virtual server from one type to another, there may be changes automatically applied to some of the settings. If you change the type back to its original value, those changes remain, and are saved when you click Update.
Conditions:
-- Modifying a virtual server from one type to another, and then changing it back to the original type.
-- Clicking Update.
Impact:
Unexpected configuration changes, which can lead to unexpected behavior of the BIG-IP system.
Workaround:
To prevent unwanted changes, when you change a virtual server's type and then change it back within the same session, click Cancel instead of Update.
Component: Access Policy Manager
Symptoms:
'Class Attribute' value in the Radius-Acct STOP request from the BIG-IP APM system does not match the 'Class Attribute' value in the Radius-Acct START request from the RADIUS server.
Conditions:
-- Access policy is configured with RADIUS Acct VPE item to send accounting messages to RADIUS server when users log on and off.
Impact:
RADIUS server does not log accounting information properly.
Workaround:
This workaround textually describes reconfiguring an access policy in the Visual Policy Editor (VPE). Descriptions may not be as straightforward as in regular GUI workarounds.
This is a description of the visual layout of the policy:
Start --+-- Logon Page --+-- RADIUS Auth --+-- RADIUS Acct --+-- Variable Assign (1) --+-- Advanced Resource Assign --+-- Variable Assign --+-- Allow
1. Decode the class attribute and save it in a temporary variable:
-- Click Variable Assign (1); in the two sub-areas, enter the following:
temp.radius.last.attr.class.decoded
expr { [mcget -decode {session.radius.last.attr.class}] }
2. Copy the temporary variable into the class session var:
-- Click Variable Assign; in the two sub-areas, enter the following:
session.radius.last.attr.class,
expr { [mcget {temp.radius.last.attr.class.decoded}] }
In this way, when you log out, and APM sends the RADIUS Stop accounting message, it uses the decoded value that is saved in the session.radius.last.attr.class variable.
Component: TMOS
Symptoms:
When creating a Rate Shaping Rate Class in the GUI, the default values for 'Queue Method' and 'Drop Policy' are not correct.
Conditions:
-- Creating a Rate Shaping Rate Class in the GUI.
-- Leaving 'Queue Method' and 'Drop Policy' settings as their defaults.
Impact:
Unexpected values in the configuration: 'Queue Method' is 'sfq' and 'Drop Policy' is 'tail'.
Workaround:
You can use either of the following workarounds:
-- Manually set the 'Queue Method' and 'Drop Policy' when creating a Rate Shaping Rate Class object. These settings are available in the 'Advanced' view of the 'General Properties' section. 'Queue Method' should be 'pfifo' and 'Drop Policy' should be 'fred'.
-- Use TMSH to create Rate Shaping Rate Class objects.
Component: Local Traffic Manager
Symptoms:
The WebSocket profile allows use of the WS::disconnect iRule command to gracefully terminate a connection with a client or a server. Use of this command may result in crash if tmm parts the iRule before execution completes.
Conditions:
-- BIG-IP has a virtual server configured with a WebSocket profile.
-- An iRule the includes the WS::disconnect command is attached to the virtual server.
-- BIG-IP is under heavy load and/or the iRule requires an extended time to execute, which might happen, for example, during execution on an iRule, tmm might park the iRule execution because the operation takes more CPU cycle than tmm can allocate to complete the iRule execution.
Impact:
Traffic disrupted while tmm restarts.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
When rate shaping is configured, the system might suppress TCP RSTs issued by itself.
Conditions:
Rate shaping is configured.
Impact:
The rate-shaping instance drops TCP RSTs; the endpoint is not informed about the ungraceful shutdown.
Workaround:
Do not use rate-shaping.
Component: Application Security Manager
Symptoms:
A brute force attack coming from an entity (such as an IP address, etc.) may be stopped prematurely.
Conditions:
-- Many brute force attacks are happening at once, coming from many sources.
-- Distributed attack is not detected (due to configuration).
Impact:
At some point, an entity might not be mitigated due to the sheer number of mitigated entities. When this occurs, there is no notification.
Workaround:
None.
Component: Application Security Manager
Symptoms:
Live Update default factory file for Server Technologies cannot be reinstalled once it is no longer the currently installed update file.
Conditions:
This occurs:
-- Once another update file for Server Technologies has been installed (most likely, a newer file).
-- If the device has been upgraded from a prior release such that the currently installed Server Technologies file is from the previous release, and is not the default factory file for the current release.
Impact:
Live Update default factory file for Server Technologies cannot be reinstalled.
Workaround:
None.
Component: TMOS
Symptoms:
On the Network :: Tunnels page of the GUI, if the Tunnel Static Forwarding Table contains more than one page of data, you are unable to advance past the first page.
Conditions:
Tunnel Static Forwarding Table contains a large number of configured tunnels.
Impact:
The content displayed on screen and does not conform to the system preference for Records Per Screen.
Workaround:
None.
Component: TMOS
Symptoms:
If ICAP is not licensed, the system does not prevent you from adding Request Adapt Profiles and Response Adapt Profiles to virtual servers in the CLI. If these profiles are configured, traffic does not pass through the virtual server and the following error is reported in /var/log/ltm:
crit tmm[3328]: 01010022:2: ICAP feature not licensed
Conditions:
-- ICAP is not licensed.
-- At least one virtual server has been configured with a Request Adapt Profile and/or a Response Adapt Profile.
Impact:
Traffic does not pass through the affected virtual servers.
Workaround:
Remove any configured Request Adapt Profiles and Response Adapt Profiles from virtual servers.
Component: TMOS
Symptoms:
The NSM process might enter a state where it gets stuck at 100% CPU usage.
Conditions:
ECMP routes reachable via recursive nexthops.
Impact:
NSM is stuck at 100% CPU usage.
Workaround:
Avoid using EMCP routes reachable via recursive nexthops.
Component: TMOS
Symptoms:
The Request Adapt Profile and Response Adapt Profile settings are visible when creating or editing a virtual server in the GUI on systems that do not have ICAP licensed. If these profiles are configured, traffic does not pass through the virtual server and the following error is reported in /var/log/ltm:
crit tmm[3328]: 01010022:2: ICAP feature not licensed
Conditions:
-- ICAP is not licensed.
-- At least one virtual server has been configured with a Request Adapt Profile and/or a Response Adapt Profile.
Impact:
Traffic does not pass through the affected virtual servers.
Workaround:
Remove any configured Request Adapt Profiles and Response Adapt Profiles from virtual servers.
Component: Advanced Firewall Manager
Symptoms:
PCCD enters a restart loop, until the configuration is changed such that 256 or fewer FQDN entries are in use.
Conditions:
Greater than 256 FQDN entries are in use in Firewall Rules.
Impact:
PCCD restart loop. PCCD is not functional until there are 256 or fewer entries.
Workaround:
Use 256 or fewer FQDN entries in Firewall Rules.
To aid in the removal of extra rules when using tmsh, you can prevent PCCD restart messages from flooding the console:
1. Stop PCCD to halt the restart messages:
bigstart stop pccd
2. Modify the configuration.
3. Bring PCCD back up:
bigstart start pccd
Component: Global Traffic Manager (DNS)
Symptoms:
DB validation exception occurs during sync or config load:
01070734:3: Configuration error: DB validation exception, unique constraint violation on table (gtm_wideip_alias) object ID (1 /Common/alias.test.com www.test.com). A duplicate value was received for a non-primary key unique index field. DB exception text (Cannot update_indexes/checkpoint DB object, class:gtm_wideip_alias status:13)
Unexpected Error: Loading configuration process failed.
Conditions:
-- Wide IP has an alias associated with it.
-- Sync or load the config.
Impact:
You are unable to load config or full sync from peer GNS/GTM.
Workaround:
Follow this procedure:
1. Delete the wide IP alias on the destination device.
2. Try the sync or load config operation again.
Component: Global Traffic Manager (DNS)
Symptoms:
During a topology load balancing decision, TMM can crash.
Conditions:
-- Topology records are deleted.
-- A load balancing decision using topology load balancing occurs.
Impact:
On very rare occasions, TMM can crash. Traffic disrupted while tmm restarts.
Workaround:
None.
Component: TMOS
Symptoms:
If a BIG-IP device has more than 24 tmm instances and one of the tmm processes above the 24th cpu loops (e.g., in response to an internal issue), it loops indefinitely.
Conditions:
-- BIG-IP i15600 / i15800 platforms.
-- Another issue occurs that that causes a tmm process greater than the 24th tmm process to loop.
Impact:
Traffic disrupted on the tmm process that is looping indefinitely.
Workaround:
1. Manually change /defaults/daemon.conf to include the appropriate tmm number and respective heartbeat action if the supported tmm is not listed.
Note: The change does not persist across software installs.
a. mount -o remount,rw /usr
b. Edit /defaults/daemon.conf and put these contents at the top of the file:
sys daemon-ha tmm24 {
description none
heartbeat enabled
heartbeat-action go-offline-downlinks-restart
running enabled
running-timeout 2
sys daemon-ha tmm25 {
description none
heartbeat enabled
heartbeat-action go-offline-downlinks-restart
running enabled
running-timeout 2
sys daemon-ha tmm26 {
description none
heartbeat enabled
heartbeat-action go-offline-downlinks-restart
running enabled
running-timeout 2
sys daemon-ha tmm27 {
description none
heartbeat enabled
heartbeat-action go-offline-downlinks-restart
running enabled
running-timeout 2
c. mount -o remount,ro /usr
2. After performing the edit, load the changes into the running configuration via 'tmsh load sys config partitions all'.
3. Verify that sod is now correctly monitoring tmm instances above tmm24 using a command such as:
tmsh show sys ha-status all-properties | grep "daemon-heartbeat" | grep tmm
Component: Local Traffic Manager
Symptoms:
When a pool member is within its slow-ramp period, and is a member of a pool that uses a static-ratio-based load balancing algorithm, its ratio weight is not taken into account when balancing connections to it. If it has a ratio that is higher than other pool members, this can result in a sudden influx of connections once the pool member exits the slow-ramp period.
Conditions:
-- Pool with a non-zero slow-ramp timeout and a static-ratio-based load balancing algorithm.
-- Pool members within the pool have different ratio weights.
-- At least one pool member is inside its slow-ramp period.
Impact:
The pool member could still be overwhelmed despite the attempt to slow-ramp connections to it.
Workaround:
None.
Component: Application Security Manager
Symptoms:
When Bot Defense performs a 307 Redirect, the cookie is set on the URL if Bot Defense detects the Safari browser.
Conditions:
-- Bot Defense profile is attached to virtual server.
-- 'Browser Verification' set to 'Verify Before Access' or 'Verify After Access'.
-- 'Cross Domain Requests' set to 'Validate Upon Request'.
-- Surfing on Safari browser to a related domain.
Impact:
A cookie is set on the URL.
Workaround:
None.
Component: Access Policy Manager
Symptoms:
The default value of 'Enable Always Connected' becomes enabled for connectivity profiles after adding 'Server List' to one of the connectivity profiles with the same Parent profile.
Conditions:
-- Add 'Server List' to one of the connectivity profiles with the same Parent profile.
-- Enable the 'Always Connected mode' setting for one of the connectivity profiles.
Impact:
This change affects all of the connectivity profiles. The parent-child inheritance logic is broken.
Workaround:
Manually uncheck the setting in all new profiles where 'Enable Always Connected' is not needed.
Component: TMOS
Symptoms:
A message is logged with a very large wchan (waiting channel, WCHAN :: Sleeping in Function) value that corresponds to -1 when read as signed instead of unsigned.
Conditions:
This happens in normal operation.
Impact:
The message is not accurately reporting the wchan value
Workaround:
Look at the /proc/PID/stack file for the correct wchan value.
Component: Service Provider
Symptoms:
Tmm crashes while passing traffic.
Conditions:
-- Per-request policies configured.
-- ICAP is configured.
This is rare condition that occurs intermittently.
Impact:
Traffic disrupted while tmm restarts.
Workaround:
None.
Component: Service Provider
Symptoms:
Validation error is observed while attaching DoS security profile to GPRS Tunneling Protocol (GTP) virtual server.
Conditions:
Attach DoS security profile to GTP virtual server.
Impact:
Validation error. Cannot attach DoS profile to GTP virtual server.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
Running 'tmsh install sys crypto key' command for SafeNet keys creates a new mcp object with the keyname as the label name in HSM, and with the duplicate key-id.
Conditions:
This happens when trying to install the key using the key-label as the argument.
Impact:
Even after deleting the tmsh key (tmsh delete sys crypto key), the BIG-IP system can still pass traffic because there's a duplicate key pointing to the same key in the HSM.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
The connection close reason in QUIC is limited to 5 bytes prior to the path being validated. This makes it difficult to debug early connection failures.
Conditions:
A connection close is sent in QUIC before the path is validated.
Impact:
Connection close reason is limited to 5 bytes before the path is validated.
Workaround:
Component: TMOS
Symptoms:
With LDAP as authentication provider, some administrative users are unable to login to BIG-IP.
Conditions:
-- LDAP remote authentication.
-- Certain users, whose characteristics have not been identified.
Impact:
Certain client users cannot login to administer the BIG-IP system.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
An assert is reported in the logs:
panic: ../net/ifc.c:975: Assertion "ifc ref valid" failed.
Conditions:
The conditions under which this occurs are unknown.
Impact:
Tmm crashes and restarts. Traffic disrupted while tmm restarts.
Workaround:
None.
Component: TMOS
Symptoms:
When using the pool statistics GUI page, the page stops displaying and the GUI shows the following error:
General database error retrieving information.
Conditions:
You attempt to apply a Status filter (e.g., Available) to display only some pools.
Impact:
The Status filter is not usable. Additionally, the page continues not to display even after you navigate away from the page and later return to it.
Workaround:
There is no workaround to prevent the issue, but if you wish to access that page again (and not use the Status filter), you can do so by clearing your browser's cache.
Component: Access Policy Manager
Symptoms:
After enabling 'show-extended-error' on the RADIUS Auth agent, instead of seeing the expected message: 'The username or password is not correct. Please try again.', the system reports the message: (error: Access-Reject).
Conditions:
RADIUS Auth with 'show-extended-error' enabled.
Impact:
The content of the Reply Message is not reported. The actual reported error message is confusing and provides no assistance in resolving the condition causing the access error: username, password, passcode, or tokencode.
Workaround:
None.
Component: TMOS
Symptoms:
Running the imish command hangs when ospfd is enabled.
Conditions:
-- Dynamic routing enabled.
-- The ospfd protocol is enabled.
-- Running the imish command.
Impact:
The imish operation hangs.
Workaround:
Restart the ospfd daemon.
Component: Anomaly Detection Services
Symptoms:
Virtual Server Health should not take into account deliberate drop requests.
Conditions:
-- DoS profile is attached to Virtual Server.
-- iRule that drops requests on some condition is also attached to the virtual server.
Impact:
Server Health reflects it is overloading status more precisely.
Workaround:
Do not use iRules to drop requests when Behavioral DoS is configured.
Component: TMOS
Symptoms:
Navigating to Statistics -> Dashboard in the GUI with AVR or APM provisioned causes continuous audit logging and restjavad logs.
Conditions:
-- AVR provisioned
-- An administrator navigates to Statistics -> Dashboard.
Impact:
The continuous extra logs will lead to confusion and may not be helpful
Component: TMOS
Symptoms:
Dynamic routing changes do not cause point-to-point tunnel flows to refresh their connection entries causing tunneled traffic to be dropped/discarded.
Conditions:
Path to a remote tunnel endpoint is provided by a dynamic routing.
Impact:
Tunneled traffic might be dropped/discarded by the BIG-IP system.
Workaround:
Use static routing to provide a path to remote tunnel endpoint.
Component: Advanced Firewall Manager
Symptoms:
Excessive and large DoS debug messages associated with tmsh commands and stat queries are logged to /var/tmp/mcpd.out which is not log-rotated.
Conditions:
-- AFM is provisioned.
-- DoS queries executed via tmsh.
Impact:
Disk space is consumed on the filesystem for /var/tmp, which can eventually lead to follow-on failures when the disk fills up.
Workaround:
Delete or purge /var/tmp/mcpd.out.
Component: TMOS
Symptoms:
Availability status of virtual server can be left 'available' even if the corresponding pool's availability becomes 'unknown'.
Conditions:
- Pool member is configured as an FQDN node.
- You set monitor to 'none' with the pool.
Impact:
Inconsistent availability status of pool and virtual server.
Workaround:
Set the FQDN node to 'force offline', and then 'enable'. This triggers virtual server's status updates and syncs to pool.
Component: Application Security Manager
Symptoms:
When the end server application hooks some of the XHR callback functions (onload, onreadystate, send, open..), and a Single Page Application is used, the application page may malfunction.
Conditions:
This may occur when a server application hooks some XHR callbacks, and one of the following:
-- Bot Defense Profile with 'Single Page Application' enabled and attached to the virtual server.
-- ASM Policy with 'Single Page Application' enabled and a JS feature is attached to the virtual server.
-- DoS Profile with Application Security and 'Single Page Application' enabled and is attached to the virtual server.
Impact:
Web page does not render properly. It might result in a blank page, missing functionality, etc.
Workaround:
Run the command:
tmsh modify sys db dosl7.parse_html_inject_tags value "after,body"
Component: Application Visibility and Reporting
Symptoms:
If avrd fails a core file created in this case is named accordung to the thread name and has no indication that it belongs to avr, for example: SENDER_HTTPS.bld0.0.9.core.gz
Conditions:
Avrd fails with a core
Impact:
It is inconvenient for identifying the process that caused the core.
Component: TMOS
Symptoms:
After attempting to assign an LTM policy to a virtual server, you may get an error in the GUI:
010716d8:3: Compilation of ltm policies for virtual server /Common/VS failed; Couldn't publish shared memory segment.
You may also see this logged in /var/log/ltm:
err mcpd[6905]: 010716d5:3: Failed to publish LOIPC object for (loipc_VS.1589526000.777142413). Call to (shm_open) failed with errno (13) errstr (Permission denied).
Conditions:
-- Run the command:
mount -a
-- Attempt to assign an LTM policy to a virtual server.
Impact:
Unable to apply LTM policies.
Workaround:
Run the following command to enable assigning an LTM policy to a virtual server:
umount -l /dev/shm
Component: Local Traffic Manager
Symptoms:
VLAN Failsafe timeout is triggered later than its configured interval.
Conditions:
-- Rarely happen on first failover. More commonly occurs on 3rd/4th failover.
-- Specific conditions that cause this issue are not known.
Impact:
VLAN Failsafe timeout might be triggered later than it is configured.
The exact impact varies based on the configuration and traffic activity.
Workaround:
Use another form of automatic failover if needed (gateway failsafe, ha-groups, etc.).
Component: Anomaly Detection Services
Symptoms:
Tmm crash when using iRule to reject connections when Behavioral DoS is enabled.
Conditions:
This can occur due to an interaction between a Behavioral DoS policy and an iRule designed to potentially drop some of the connections.
Impact:
With heavy traffic, the tmm process might crash. Traffic disrupted while tmm restarts.
Workaround:
Do not use iRules to reject connections that are bound to a virtual server with a Behavioral DoS policy attached.
Component: Application Security Manager
Symptoms:
When a Cloud Auto Scaling deployment is set up using F5's Auto Scale Template, and ASM Live Update is configured with Automatic Download enabled, Live Update configuration may be overwritten during a scale out event when a new host joins the sync cluster. This is caused by a config sync from the new device to the master device, before the master has a chance to sync the configuration to the new device, causing the configuration in the master device to be overwritten.
Conditions:
-- Using F5's Auto Scaling template.
-- Auto Scale script is configured with --block-sync (which is the default).
-- ASM Live Update is configured with Automatic Download enabled.
-- A scale out event occurs.
-- New ASU is automatically downloaded by Live Update at the new host.
Impact:
Live Update configuration of all devices in the Auto Scale group is overwritten.
Workaround:
Disable ASM Live Update Automatic Download.
This can be done by disabling the liveupdate.autodownload DB variable using the onboard.js script, and adding '-d liveupdate.autodownload:disable'.
For example:
/usr/bin/f5-rest-node /config/cloud/aws/node_modules/@f5devcentral/f5-cloud-libs/scripts/onboard.js --log-level silly --signal ONBOARD_DONE -o /var/log/cloud/aws/onboard.log --host localhost --port 8443 -d tm.tcpudptxchecksum:software-only -d liveupdate.autodownload:disable --ping
-d tm.tcpudptxchecksum:software-only -d liveupdate.autodownload:disable
In order to still have automatic updates for the group, the db variable can be enabled for the master device. Then this setting will be applied on every new host after joining the group and receiving the initial sync from the master.
Component: TMOS
Symptoms:
BIG-IP Engineering Hotfixes may not show the summary titles for fixed bugs (as appear for the affected bugs published via Bug Tracker).
-- The 'tmsh show sys version' command displays the bug numbers for fixes included in Engineering Hotfixes.
-- If a given bug has been published via Bug Tracker, the summary title of the bug is expected to be displayed as well.
-- Running BIG-IP Engineering Hotfixes built on or after March 18, 2019.
Conditions:
For affected BIG-IP Engineering Hotfixes, titles are not displayed for any bugs fixed in the Engineering Hotfix.
Impact:
Cannot see the summaries of the bugs fixed by running the 'tmsh show sys version' command.
Workaround:
For bugs that are published via Bug Tracker, you can query for the affected bug in Bug Tracker (https://support.f5.com/csp/bug-tracker).
Note: Not all bugs fixed in BIG-IP Engineering Hotfixes are published to Bug Tracker.
For information on such bugs, consult F5 support, or the original Service Request submitted to F5 in which the affected Engineering Hotfix was requested.
Component: Global Traffic Manager (DNS)
Symptoms:
You are unable to create a backup UCS.
You see a warning in /var/log/restjavad.0.log:
[WARNING][8100/tm/shared/sys/backup/306b4630-aa74-4a3d-af70-0d49bdd1d89e/worker UcsBackupTaskWorker] Failure with backup process 306b4630-aa74-4a3d-af70-0d49bdd1d89e.
This is followed by a list of some files in /var/named/config/namedb/.
Conditions:
Named has some Slave zones configured and is going through frequent zone transfer.
Impact:
You are unable to create a UCS file.
Workaround:
Stop named zone transfer while doing UCS backup.
Component: TMOS
Symptoms:
Once per minute, syslog-ng logs nothing for 20 seconds.
Conditions:
-- A remote syslog server is specified by hostname, forcing syslog-ng to resolve it.
-- the DNS resolution times out (for example, if the DNS server is unreachable)
Impact:
When using DNS names to specify remote syslog destinations and DNS is unreachable, syslog-ng re-attempts to resolve the name every 60 seconds. This resolution has a 20 seconds timeout, and blocks the syslog process from writing logs to disk during that time.
Note that the logs are buffered, not lost, and will still be written to disk (with the correct timestamps) once the DNS query times out.
Workaround:
None.
Component: TMOS
Symptoms:
Traffic imbalance between tmm threads. You might see the traffic imbalance by running the following command:
tmsh show sys tmm-traffic
Conditions:
Source ports used to connect to i15000, i15800, i5000, i7000, i10000, i11000 and B4450 blades form an arithmetic sequence.
For example, some client devices always use even source port numbers for ephemeral connections they initiate. This means the 'stride' of the ports selected is '2'. Because a sorted list of the ports yields a list like 2, 4, 6, 8... 32002, 32004. It is 'striding' over the odd ports; thus, a port stride of 2.
Impact:
Traffic imbalance may result in tmm threads on different CPU cores having imbalanced workloads. While this can sometimes impact on performance, an overloaded tmm thread can usually redistribute load to less loaded threads in a way that does not impact performance. However the loads on the CPU cores will appear imbalanced still.
Workaround:
Where possible, configure devices to draw from the largest possible pool of source ports when connecting via a BIG-IP system.
Component: Application Security Manager
Symptoms:
The system reports an error message when trying to navigate to 'Security :: Policies' under virtual server properties:
An error has occurred while trying to process your request.
Conditions:
-- An FTP or SMTP profile with protocol security enabled is attached to a virtual server.
-- Attempt to navigate to 'Security :: Policies'.
Impact:
-- No policies appear. You cannot perform any operations on the 'Security :: Policies' screen.
-- The following error message appears instead:
An error has occurred while trying to process your request.
Workaround:
As long as an FTP or SMTP profile with protocol security enabled is defined under virtual server properties (in another words, it is attached to a virtual server), this issue recurs. There are no true workarounds, but you can avoid the issue by using any of the following:
-- Use another profile, such as HTTP.
-- Set the FTP/SMTP profile under the virtual server settings to None.
-- Remove the profile via the GUI or the CLI (e.g., you can remove the profile from the virtual server in tmsh using this command:
tmsh modify ltm virtual /Common/test-vs { profiles delete { ftp_security } }
Component: TMOS
Symptoms:
Tmm and mcpd cores for slot2
Conditions:
On the standby chassis, reboot the primary blade and wait for it to rejoin the cluster. Once it does, mcpd will core.
Impact:
Traffic disrupted while tmm and mcpd mcpd restarts.
Workaround:
Component: TMOS
Symptoms:
When REST API PUT request is call to modify LTM data-group internal without 'type' field in body-content, it fails intermittently with a 400 error.
--{"code":400,"message":"invalid property value \"type\":\"\"","errorStack":[],"apiError":26214401}
The 'type' field is not getting populated with default value and is set to null string "" instead.
Conditions:
-- PUT REST API request used to modify LTM data-group internal.
-- Type field is not specified in the body-content.
Impact:
Unable to update the configuration object (LTM data-group internal) with REST API PUT.
Workaround:
Include 'type' field inside PUT request body content for the operation to succeed.
Example Curl Command:
-- curl -isku <username>:<password> -H "Content-Type: application/json" -X PUT -d '{"records":[{"name":"1.1.1.1"}, {"name":"1.1.1.2"}], "type":"ip"}' https://localhost/mgmt/tm/ltm/data-group/internal/~Common~<Name>
Component: Traffic Classification Engine
Symptoms:
The webroot daemon (wr_urldbd) cores.
Conditions:
This can occur while passing traffic when webroot is enabled.
Impact:
The wr_urldbd daemon cores. URL Categorization functionality may not work as expected.
Workaround:
None.
Component: TMOS
Symptoms:
When a blade fails, other blades may try to forward traffic using trunked interfaces on the down blade.
Conditions:
-- A multi-blade chassis.
-- Interfaces are trunked.
-- A blade is pulled or powered off.
Impact:
Some traffic is dropped until the failed blade is detected by clusterd (10 seconds by default.)
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
While deleting iFiles, TMM might crash and generates a core.
Conditions:
-- There is any ifile command in an iRule where you are giving a list object to the file name argument instead of to a string object, for example:
The 2nd line returns a list object, and the 3rd line is giving it to ifile command:
set ifile_name tcl.1
set ifile_name [lsearch -glob -inline [ifile listall] "*$ifile_name"]
ifile get $ifile_name
-- A request is processed by the iRule.
-- Attempt to delete the file in which the issue occurred.
Impact:
TMM crashes and restarts, triggering failover in high availability (HA) configurations. Traffic disruption while TMM restarts on a standalone configuration.
Workaround:
Ensure that the iRule gives a string object to ifile commands. Here is a sample iRule that implements the workaround:
set ifile_name tcl.1
set ifile_name [lsearch -glob -inline [ifile listall] "*$ifile_name"]
ifile get [format "%s" $ifile_name]
Component: Service Provider
Symptoms:
-- Unable to establish MRF connection after failover.
-- Error reports 'no connection'.
Conditions:
- MRF configured.
- Using iRule for routing.
-- Failover occurs.
Impact:
Unable to establish new connection until existing sessions time out. No message is reported explaining the circumstances.
Workaround:
Any of the following:
-- Enable connection mirroring on the virtual server.
-- Disable session mirroring.
Component: Application Visibility and Reporting
Symptoms:
When the avrd process is stopped or restarted, it fails with core before the exit. A core file with the name starting with SENDER_HTTPS (for example, SENDER_HTTPS.bld0.0.9.core.gz) can be found in /shared/cores/ directory.
Conditions:
A BIG-IP system is registered on BIG-IQ and has established an HTTPS connection with BIG-IQ for sending stats data.
Impact:
Avrd cores while exiting. There is no impact on BIG-IP system functionality.
Workaround:
None.
Component: Global Traffic Manager (DNS)
Symptoms:
Different link throughput statistics are seen on GTM/DNS systems that are connected by full-mesh iQuery.
Conditions:
-- The same link is used on different BIG-IP addresses as a pool member in the default gateway pool.
-- A forwarding virtual server is used.
Impact:
Each GTM/DNS server might get different link throughput for the same link, and therefore make less-than-optimal decisions.
Workaround:
Do not use the same uplink for different BIG-IP devices.
Component: Local Traffic Manager
Symptoms:
If an LTM database monitor type (MySQL, MSSQL, Oracle, or PostgreSQL database monitor type) is configured with a 'send' string but with no 'receive' string to issue a user-specified database query, pool members using this monitor are marked DOWN, even though a connection to the configured database completed successfully.
Conditions:
-- An LTM pool or pool members is configured to use an LTM database (MySQL, MSSQL, Oracle or PostgreSQL) monitor type.
-- A 'send' string is configured for the monitor.
-- A 'receive' string is not configured.
Impact:
The database monitor marks the pool member down, even in cases where the pool member is actually pingable.
Workaround:
To work around this issue, configure 'send' and 'recv' strings for the database monitor that will always succeed when successfully connected to the specified database (with the configured username and password, if applicable).
Component: Local Traffic Manager
Symptoms:
TMM crash.
Conditions:
-- Modify in-tmm monitors.
-- Perform configuration sync.
Impact:
Traffic disrupted while tmm restarts.
Workaround:
-- Disable in-tmm monitors.
-- Avoid config syncing to the active device.
Component: Local Traffic Manager
Symptoms:
-- Connections to the virtual server might hang.
-- Increased tmm CPU utilization.
Conditions:
-- A virtual server is configured with a traffic-matching-criteria that utilizes a source-address-list and/or destination-address-list.
-- The virtual server utilizes certain persistence one of the following persistence types:
+ Source Address (but not hash-algorithm carp)
+ Destination Address (but not hash-algorithm carp)
+ Universal
+ Cookie (only cookie hash)
+ Host
+ SSL session
+ SIP
+ Hash (but not hash-algorithm carp)
Impact:
-- High tmm CPU utilization.
-- Stalled connections.
Workaround:
Enable match-across-virtuals in the persistence profile.
Note: Enabling match-across-virtuals might might affect the behavior of other virtual servers in the configuration that utilize persistence.
Component: Application Security Manager
Symptoms:
Certain roles, such as Resource Administrator and Application Security Operations Administrator, do not have sufficient permission levels to perform Live Update.
Conditions:
-- User with Resource Administrator or Application Security Operations Administrator role assigned.
-- Attempt to perform Live Update.
Impact:
Users with Resource Administrator and Application Security Operations Administrator role cannot perform Live Update.
Workaround:
None.
Component: Global Traffic Manager (DNS)
Symptoms:
When using DNS Cache on chassis systems with a forward zone pointing at a self IP for communication with local BIND, the following assert triggers:
tmm_panic (... "../net/loop.c:572: %sIDX set on listener%s") at ../lib/stdio.c:1307
Conditions:
-- Chassis system is used.
-- Secondary TMMs core dump.
-- Primary works as expected.
Impact:
TMMs on secondary blades core dump. Traffic disrupted while tmm restarts.
Workaround:
1) Create another virtual server that uses local Bind server.
2) Set the forward zones to point to that virtual server instead of the self IP as name servers.
Component: Application Security Manager
Symptoms:
Policy Builder is issuing a learning suggestion to set a specific maximum length for a parameter when that parameter already has that exact maximum length already configured.
Conditions:
-- Response learning is turned on
-- Response parameter length is less than, but close to, the currently configured maximum length limit.
Impact:
Redundant learning suggestion is issued.
Workaround:
You can either:
-- Ignore the learning suggestion (Click the Ignore button).
-- Turn off Learn from response.
Component: Policy Enforcement Manager
Symptoms:
PEM sessions go to a stale state and the Credit Control Request (CCRi) is not sent.
Conditions:
-- PEM is configured and passing normal PEM traffic.
-- Using BIG-IP Virtual Edition (VE)
Impact:
Session is not established.
Workaround:
None.
Component: Global Traffic Manager (DNS)
Symptoms:
The iqsyncer utility leaks memory.
Conditions:
-- There is a large bigip_gtm.conf.
-- The log.gtm.level is set to debug.
Impact:
The iqsyncer utility exhausts memory and is killed.
Workaround:
Do not set log.gtm.level equal to or higher than debug.
Component: Local Traffic Manager
Symptoms:
An iRule executing on the FLOW_INIT event can suspend. If it does so while connecting to a virtual-to-virtual flow, it can cause a TCP crash, which results in a tmm restart.
Conditions:
An iRule executing on the FLOW_INIT event suspends while connecting to a virtual-to-virtual flow.
Impact:
Tmm crashes. Traffic disrupted while tmm restarts.
Workaround:
Do not include any iRules that suspend processing in FLOW_INIT.
Component: Local Traffic Manager
Symptoms:
Whenever a new multicast entry is added to an already full table, an error is logged to /var/log/tmm
Conditions:
This warning occurs when the multicast table is full
Impact:
When this condition occurs, this can fill the tmm log with these warning messages
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
A tmm core occurs unexpectedly and causes a failover event.
Conditions:
An internal error occurs when deleting an old SSL session during SSL handshake.
There are no externally facing conditions.
Impact:
No known impact.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
Thales installation script fails with error message.
ERROR: Could not reach Thales HSM "<ip>". Make sure the HSM IP address is correct and that the HSM is accessible to the BIG-IP.
Conditions:
This occurs when the ICMP ping is blocked between the BIG-IP system and netHSM.
Impact:
Thales/nCipher NetHSM client software installation fails.
Workaround:
Unblock ICMP ping between the BIG-IP system and netHSM.
Component: Local Traffic Manager
Symptoms:
If an iRule utilizing the clientside or serverside command causes parking to occur while in the clientside/serverside command (table or after commands, for example), the connection is aborted while parked, and a subsequent iRule event attempts to run (CLIENT_CLOSED, for example), tmm may restart.
Conditions:
-- iRule using clientside or serverside command.
-- Use of commands that may park while in the clientside/serverside command.
-- Flow is aborted while iRule is parked.
-- iRule also has CLIENT_CLOSED or SERVER_CLOSED event handler.
For more information on the conditions that trigger iRule parking, see K12962: Some iRules commands temporarily suspend iRules processing, available at https://support.f5.com/csp/article/K12962.
Impact:
Traffic disrupted while tmm restarts.
Workaround:
You can use either of the following workarounds:
-- Rework iRules to avoid use of clientside/serverside command.
-- Eliminate parking within the clientside/serverside commands.
Component: TMOS
Symptoms:
After upgrading BIG-IP APM, multiple error messages appear in /var/log/ltm:
-- err mcpd[5352]: 010713cf:3: Parsing default XML files. Failed to parse xml file (/var/sam/www/client/customization-source/Common/modern/secure_access_client/default_secure_access_client.xml) because Failed to stat file (/var/sam/www/client/customization-source/Common/modern/secure_access_client/default_secure_access_client.xml) errno(2) strerror(No such file or directory)
-- err mcpd[5352]: 010713cf:3: Parsing default XML files. Failed to parse xml file (/var/sam/www/client/customization-source/Common/modern/resource_app_tunnel/default_resource_app_tunnel.xml) because Failed to stat file (/var/sam/www/client/customization-source/Common/modern/resource_app_tunnel/default_resource_app_tunnel.xml) errno(2) strerror(No such file or directory)
Conditions:
-- APM configuration.
-- Upgrade the BIG-IP system to v15.1.0 or newer.
Impact:
These are benign messages that do not indicate a functional issue. There is no impact; the system works correctly.
The errors occur when the upgraded BIG-IP APM configuration attempts to load resource definitions for the modern customization schema. However, by design, the modern customization schema does not define resources. Only the standard customization schema defines resources found under '/var/sam/www/client/customization-source/Common/standard/'.
Workaround:
None.
Component: TMOS
Symptoms:
There is no FPGA support for vector number 105 (FPGA vector number). This is an L4 DDoS vector that rate limits the number of incorrect TCP ACK Cookies. The vector is commonly referred to as ACK_Cookie_Bad.
Conditions:
This is encountered when a TCP BADACK DDoS vector is detected.
Impact:
This vector is not hardware accelerated. The DDoS mitigation can rely only on software support for this DDoS vector.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
In GUI an SSL certificate's version is displayed as '1', even if its version is higher than 1.
Conditions:
-- Viewing an SSL certificate in the GUI.
-- The SSL certificate's version is higher than 1.
Impact:
SSL certificate's version is displayed as '1'. There is no functional impact.
Workaround:
None.
Component: Application Security Manager
Symptoms:
After upgrade, some requests can cause BD errors on response:
BEM|ERR |May 19 17:49:55.800|0983|response_header_accumulator.c:0200|Error: CookieMgrBuildCookie failed. ans 1 job 2957561040
IO_PLUGIN|ERR |May 19 17:49:55.800|0983|io_plugin.c:3320|TMEVT_RESPONSE: Cannot build a ts cookie.
Conditions:
-- Upgrading BIG-IP systems to v15.0.0 or later from versions earlier than v15.0.0.
-- ASM policy is configured on a virtual server.
Impact:
For some requests, the response can arrive truncated or not arrive at all.
Workaround:
Add an iRule that deletes ASM cookies:
when HTTP_REQUEST {
set cookies [HTTP::cookie names]
foreach aCookie $cookies {
if {$aCookie matches_regex {^TS(?:[0-9a-fA-F]{6,8})(?:$|_[0-9]+$)}} {
HTTP::cookie remove $aCookie
Note: Performing this workaround affects cookie-related violations (they may need to be disabled to use this workaround), session, and login protection.
Component: Local Traffic Manager
Symptoms:
Use of the LB::down command in an iRule may not have the desired effect.
Specifically, the pool member is marked down within the tmm thread executing the iRule, but the status change is not updated to mcpd, or to other tmm threads.
As a result, the message 'Pool /Common/mypool member /Common/1.1.1.1:80 monitor status iRule down' does not appear in the log, and the status of the pool member is not updated when viewed in the GUI or via 'tmsh show ltm pool xxxx members'.
Conditions:
Using the LB::down command in an iRule.
Impact:
Because mcpd believes the pool member to be up, it does not update tmm's status, so tmm continues to regard it as down indefinitely, or until a monitor state change occurs.
If the LB::down command is used on all members of a pool, the affected tmms cannot load balance to that pool, even though the GUI/tmsh indicate that the pool has available members.
Because pool member status is stored on per-tmm basis and incoming connections are distributed across tmms using a hash, this can lead to apparently inconsistent results, where some traffic (traffic hitting a particular tmm) is rejected with an RST cause of 'No pool member available'.
Workaround:
No direct workaround, but the use of an inband monitor instead of the LB::down command may be effective. You must tune the inband monitor's settings to values consistent with the desired behavior.
Component: Local Traffic Manager
Symptoms:
HTTP/2 allows sending the payload in both directions gradually, until a frame with an END_STREAM flag closes a direction. The BIG-IP does not properly handles an early response from a server when HTTP router is configured on a virtual and partial payload sent in each direction. In this case, communication over the stream hangs.
Conditions:
-- Virtual server is configured on the BIG-IP system with HTTP and HTTP/2 profiles on both the client and server sides.
-- HTTP router profile is configured on the virtual server.
-- Client sends a request and delivers a partial payload, waiting for a response from a server.
-- Server responds with a partial payload.
Impact:
A response with a partial payload is not delivered to a client. Communication freezes on that specific stream.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
After a rate limit is triggered and released, the virtual server status in the GUI remains as 'unavailable'. The virtual server resumes accepting new connections while the GUI shows the virtual server is unavailable.
Conditions:
-- The virtual server has a source address list configured.
-- Address lists define more than one address.
-- The connections are over the rate limit, and the virtual server status is marked unavailable.
-- The number of connections falls below the limit.
Impact:
Actual virtual server status is not reflected in GUI.
Workaround:
If the deployment design allows, you can use either of the following workarounds:
-- Remove the source address list from the virtual server.
-- Have a single address in the source address list.
Component: Local Traffic Manager
Symptoms:
When using HTTP/2, the :scheme pseudo-header appears to always be set to HTTPS on requests, even when the server-side connection is not encrypted.
Conditions:
-- Using an HTTP/2 virtual server.
-- The server-side connection that is unencrypted.
Impact:
The impact of this issue varies based on how the application reacts at the server-side.
Workaround:
None.
Component: TMOS
Symptoms:
iControl REST command to create a data group fails if you do not set the externalFileName variable.
The same command works in tmsh, and you are not required to specify the externalFileName.
Conditions:
-- Creating a data group using iControl REST.
-- POST payload does not contain the externalFileName variable.
Impact:
You are unable to create the data group.
Workaround:
The command works if you specify the externalFileName parameter:
curl -sku $PASS https://$HOST/mgmt/tm/ltm/data-group/external -X POST -H "Content-type: application/json" -d '{"name":"fooBar", "externalFileName":"fooBar.txt"}'
Component: TMOS
Symptoms:
When you delete an LTM external data-group object using iControl REST, it incorrectly returns '200 OK' even though the object is not deleted.
Conditions:
-- Deleting an external data-group object via iControl REST.
-- The LTM external data-group object is referenced by another object (such as an iRule).
Impact:
The object still exists, even though the system returns a '200 OK' message indicating that the operation completed successfully.
Workaround:
None.
Component: TMOS
Symptoms:
-- The mcpd process is killed with SIGABRT by the sod watchdog due to failed heartbeat check.
-- There is high memory usage by the mcpd process prior to getting killed.
-- There is an mcpd core file contains a very long string. The core might contain a repeating pattern of '{ } { } { } ...'.
Conditions:
The mcpd process receives a malformed message from one of the control plane daemons.
Impact:
-- There is a temporary lack of responsiveness related to actions of inspecting and/or modifying system configuration: GUI, TMSH, etc., operations may fail or time out.
-- SNMP queries might go unanswered.
-- System daemons restart.
-- Traffic disrupted while mcpd restarts.
Workaround:
None.
Component: Global Traffic Manager (DNS)
Symptoms:
In certain instances, the iqsyncer utility may not write the core file for system signals.
Conditions:
SELinux policies enabled.
-- Run the iqsyncer utilities to initiate a core dump with system signals SIGABRT, SIGQUIT, and SIGSEGV.
Impact:
The iqsyncer utility does not write core file.
Workaround:
Disable SELinux with 'setenforce 0'.
You can now generate a core with a SIGQUIT (-3), but not with SIGABRT or SIGSEGV.
Component: Global Traffic Manager (DNS)
Symptoms:
SELinux policies may prevent the iqsh/iqsyncer utilities from dumping core information.
Conditions:
-- SELinux policies enabled.
-- Run the iqsh/iqsyncer utilities to initiate a core dump.
Impact:
The operation does not result in dumping the core information.
Workaround:
Disable SELinux with 'setenforce 0'.
Component: TMOS
Symptoms:
The BIG-IP system does not prevent you from specifying previously used passwords, even if Secure Password Enforcement is enabled with password memory set to a non-zero value.
Conditions:
-- Password memory in auth settings is not 0 (zero).
-- Attempt to specify a previously specified password
Impact:
Password history to prevent user from using same password is not enforced.
Workaround:
None.
Component: TMOS
Symptoms:
When the BIG-IP system boots, mcpd continually restarts.
Conditions:
This may occur after you issue the 'diskinit' command with the '--style=volumes' option in the MOS (Maintenance Operating System) shell, install BIG-IP into the new volume, then boot into the new installation of the BIG-IP system.
Impact:
The BIG-IP system is unable to complete the boot process and become active.
Workaround:
In the MOS shell, do not issue the 'diskinit' command with the '--style=volumes' option.
Instead, on BIG-IP v14.1.2.1 and later, you may use the 'image2disk' utility with the '-format' option to recreate the desired volume.
You also can achieve the same result by following the shell guidance. To begin, type 'start<enter>'.
If the system is already in the defective state, use this shell command, and then reboot:
touch /.tmos.platform.init
The problem should be resolved.
Component: TMOS
Symptoms:
LDAP authentication fails with an error message:
err nslcd[2867]: accept() failed: Too many open files
Conditions:
This problem occurs when user-template is used instead of Bind DN.
Impact:
You cannot logon to the system using LDAP authentication.
Workaround:
None.
Component: Service Provider
Symptoms:
When an HTTP chunked request is sent to a virtual server that has both a request-adapt (e.g., for ICAP), and a request-logging profile attached, the request that is sent to the ICAP server is doubly chunked.
Conditions:
-- A virtual server is configured with both a request-adapt (ICAP) and request-logging profile.
-- HTTP chunked requests are sent to this virtual server.
Impact:
Data corruption in the HTTP stream.
Workaround:
You can use either of the following workarounds:
-- Force rechunking on the HTTP profile:
request-chunking rechunk
-- Remove the request-logging profile (and potentially log requests) using an iRule instead.
Component: Local Traffic Manager
Symptoms:
LACP system priority is calculated internally using the system MAC. This occasionally results in priority being calculated as 0, which is not accepted by third party devices, such as Cisco.
Conditions:
-- LACP is enabled on BIG-IP system.
-- The last two octets of system MAC are zero.
Impact:
LACP system priority is calculated as 0 which is rejected by Cisco.
Note: BIG-IP administrators have no manual control over setting the LACP system priority value.
Component: TMOS
Symptoms:
When a trunk is configured with a name longer than 32 characters on a vCMP host, guests update the working-mbr-count for the trunk incorrectly when another trunk on the host changes. This might result in vCMP guests failing over unexpectedly.
Conditions:
-- Trunk configured with a name longer than 32 characters on vCMP host.
-- Trunk made available to guests for high availability (HA) Group scoring.
-- At least one other trunk configured on vCMP host.
-- Interface state changes in any other trunk.
Impact:
The vCMP guests may fail over unexpectedly.
Workaround:
Do not use trunk names longer than 32 characters.
Component: TMOS
Symptoms:
The 'tmsh show sys mac-address' command indicates the management interface is using the same MAC address as a VLAN.
Conditions:
This can occur on chassis based systems and on VCMP guests. The MAC address pool does not reserve specific MAC addresses for the management interfaces and so pool entries may be reused for VLANs.
Impact:
The management MAC address is the same as the VLAN MAC address, resulting in issues relating to the inability to differentiate traffic to the management port or to traffic ports.
Workaround:
None.
Component: Access Policy Manager
Symptoms:
Authentication tab is missing in Visual Policy Editor (VPE) for RDG-RAP Access Policy type
Conditions:
Configuration of RDG-RAP Access Policy type in VPE.
Impact:
Authentication tab with AD/LDAP Query agents is missing in VPE.
Workaround:
Use the tmsh cli configuration of AD/LDAP Query for RDG-RAP Access Policy type.
Component: Application Security Manager
Symptoms:
BD crashes.
Conditions:
A specific scenario that results in memory corruption.
Impact:
Failover, traffic disturbance. Traffic disrupted while BD restarts.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
Custom iRule events are executed before the embedded APM iRule events, despite the custom iRule's priority value being larger than the APM iRule's priority value.
Conditions:
-- APM is provisioned.
-- Custom iRule with a priority value larger the APM iRule's priority value.
Impact:
Custom iRule event is executed before APM iRule event.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
QUIC does not send the HANDSHAKE_DONE immediately on conclusion of the handshake. Instead, it sends an entire flight of data first.
Conditions:
This happens in normal operation.
Impact:
The end user system has to store handshake state longer than it might have to otherwise.
Workaround:
None.
Component: Application Security Manager
Symptoms:
When trying to update Attack Signatures. the following error message is shown:
Could not communicate with system. Try to reload page.
Conditions:
Insufficient disk space to update the Attack Signature.
Impact:
Live Update unable to restore the database during startup. Device runs out of disk space, which leads to failure in writing live update hsqldb log file. The liveupdatedb.script file, which is based on the .log file, is truncated and missing necessary settings in order to initialize the live update database.
Workaround:
This following procedure restores the database to its default, initial state:
1. Remove the sigfile.tmp.* directories under /var/ts/var/tmp.
2. Delete the script:
delete /var/lib/hsqldb/live-update/liveupdatedb.script
3. Create a new script:
create new /var/lib/hsqldb/live-update/liveupdatedb.script.
4. Add the following lines to create the live update database schema and set the SA user as expected:
CREATE SCHEMA PUBLIC AUTHORIZATION DBA
CREATE MEMORY TABLE AVAILABILITY(ID VARCHAR(255) NOT NULL,ERRORMESSAGE VARCHAR(255),LASTCHECKDATETIME BIGINT,LASTCHECKUPDATEFILE VARCHAR(255),STATUS VARCHAR(255),"TYPE" VARCHAR(255),CONSTRAINT AVAILABILITY_PK PRIMARY KEY(ID))
CREATE MEMORY TABLE INSTALLSCHEDULE(ID VARCHAR(255) NOT NULL,APPLYATALLTIMES BOOLEAN,APPLYONALLDAYS BOOLEAN,APPLYONFRIDAYS BOOLEAN,APPLYONMONDAYS BOOLEAN,APPLYONSATURDAYS BOOLEAN,APPLYONSUNDAYS BOOLEAN,APPLYONTHURSDAYS BOOLEAN,APPLYONTUESDAYS BOOLEAN,APPLYONWEDNESDAYS BOOLEAN,ENDTIME VARCHAR(255),FREQUENCY VARCHAR(255),STARTTIME VARCHAR(255),"TYPE" VARCHAR(255),CONSTRAINT INSTALLSCHEDULE_PK PRIMARY KEY(ID))
CREATE MEMORY TABLE UPDATEFILE(ID VARCHAR(255) NOT NULL,CREATEDATETIME BIGINT,FILELOCATION VARCHAR(255),FILENAME VARCHAR(255),ISFILEAVAILABLE BOOLEAN,ISFILEMANUALLYUPLOADED BOOLEAN,ISGENESIS BOOLEAN,MD5 VARCHAR(255),"TYPE" VARCHAR(255),CONSTRAINT UPDATEFILE_PK PRIMARY KEY(ID))
CREATE MEMORY TABLE INSTALLATION(ID VARCHAR(255) NOT NULL,ADDEDENTITIESCOUNT INTEGER,DELETEDENTITIESCOUNT INTEGER,ERRORMESSAGE VARCHAR(255),LASTREADMEFILENAME VARCHAR(255),LASTUPDATEMICROS BIGINT,LOADDATETIME BIGINT,MODIFIEDENTITIESCOUNT INTEGER,README VARCHAR(500000),STATUS VARCHAR(255),"TYPE" VARCHAR(255),UPDATEFILE_ID_OID VARCHAR(255),CONSTRAINT INSTALLATION_PK PRIMARY KEY(ID),CONSTRAINT INSTALLATION_FK1 FOREIGN KEY(UPDATEFILE_ID_OID) REFERENCES UPDATEFILE(ID))
CREATE INDEX INSTALLATION_N49 ON INSTALLATION(UPDATEFILE_ID_OID)
CREATE MEMORY TABLE INSTALLATION_DELETEDENTITYLIST(ID_OID VARCHAR(255) NOT NULL,"ELEMENT" LONGVARBINARY,IDX INTEGER NOT NULL,CONSTRAINT INSTALLATION_DELETEDENTITYLIST_PK PRIMARY KEY(ID_OID,IDX),CONSTRAINT INSTALLATION_DELETEDENTITYLIST_FK1 FOREIGN KEY(ID_OID) REFERENCES INSTALLATION(ID))
CREATE INDEX INSTALLATION_DELETEDENTITYLIST_N49 ON INSTALLATION_DELETEDENTITYLIST(ID_OID)
CREATE MEMORY TABLE INSTALLATION_MODIFIEDENTITYLIST(ID_OID VARCHAR(255) NOT NULL,"ELEMENT" LONGVARBINARY,IDX INTEGER NOT NULL,CONSTRAINT INSTALLATION_MODIFIEDENTITYLIST_PK PRIMARY KEY(ID_OID,IDX),CONSTRAINT INSTALLATION_MODIFIEDENTITYLIST_FK1 FOREIGN KEY(ID_OID) REFERENCES INSTALLATION(ID))
CREATE INDEX INSTALLATION_MODIFIEDENTITYLIST_N49 ON INSTALLATION_MODIFIEDENTITYLIST(ID_OID)
CREATE MEMORY TABLE INSTALLATION_ADDEDENTITYLIST(ID_OID VARCHAR(255) NOT NULL,"ELEMENT" LONGVARBINARY,IDX INTEGER NOT NULL,CONSTRAINT INSTALLATION_ADDEDENTITYLIST_PK PRIMARY KEY(ID_OID,IDX),CONSTRAINT INSTALLATION_ADDEDENTITYLIST_FK1 FOREIGN KEY(ID_OID) REFERENCES INSTALLATION(ID))
CREATE INDEX INSTALLATION_ADDEDENTITYLIST_N49 ON INSTALLATION_ADDEDENTITYLIST(ID_OID)
CREATE USER SA PASSWORD ""
GRANT DBA TO SA
SET WRITE_DELAY 20
SET SCHEMA PUBLIC
5. Restart the tomcat process:
bigstart restart tomcat
Component: Advanced Firewall Manager
Symptoms:
On the AFM GUI Flow Inspector screen, there is a spelling mistake 'Additinal Info'. It should read 'Additional Info'.
Conditions:
You can locate the spelling error by following these steps:
1. Navigate to Security :: Debug :: Flow Inspector :: Get Flows (should be blank).
2. Select New Flows and then Get Flows.
3. Select the flow (i.e., click anywhere on the result except the hyperlink).
Impact:
There is a spelling mistake on the word 'Additional'. There is no functional impact to the system; this is a cosmetic issue only.
Workaround:
None.
Component: TMOS
Symptoms:
In the BIG-IP Graphical User Interface (TMUI), display of the System Menu on the LCD front panel of most BIG-IP platforms can be enabled or disabled under System :: Configuration :: Device :: General.
However, on iSeries appliances, the 'Display LCD System Menu' option does not appear on this page.
Conditions:
This occurs on the following iSeries appliances:
-- i850
-- i2000-series (i2600/i2800)
-- i4000-series (i4600/i4800)
-- i5000-series (i5600/i5800/i5820-DF)
-- i7000-series (i7600/i7600-D/i7800/i7800-D/i7820-DF)
-- i10000-series (i10600/i10600-D/i10800/i10800-D)
-- i11000-series (i11600/i11800/i11400-DS/i11600-DS/i11800-DS)
-- i15000-series (i15600/i15800)
Impact:
The 'Display LCD System Menu' option cannot be configured via the GUI.
Workaround:
You can enable display of the LCD System Menu using the Command Line (CLI) by running the following commands, in sequence:
tmsh mod sys global-settings lcd-display [enabled|disabled]
tmsh mod sys db lcd.showmenu value [enabled|disabled]
Component: TMOS
Symptoms:
The text that describes the monitor state of an LTM node, pool member, or monitor instance also contains a timestamp that initially indicates when the monitor set the affected node or pool member to the indicated state. This timestamp can be affected by other actions, such as incremental or full config sync and config load.
The monitor-state description and timestamp can be viewed in the CLI (CLI/TMSH) and GUI (TMUI) as follows:
-- From the CLI/TMSH:
tmsh show ltm monitor <monitor_type> <monitor_name>
This command shows the state of ltm nodes or pool members currently monitored by the specified ltm health monitor, as in the following example:
-------------------------------------
LTM::Monitor /Common/mysql_test
-------------------------------------
Destination: 10.10.200.28:3296
State time: down for 1hr:58mins:42sec
| Last error: No successful responses received before deadline. @2020.03.25 14:10:24
-- From the GUI:
+ Navigate to Local Traffic :: Nodes : Node List :: <node_name>. The 'Availability' field shows text describing the node's monitored state with a timestamp.
+ Navigate to Local Traffic :: Pools : Pool List :: <pool_name>, under the Members tab, click the pool member name. The 'Availability' field shows text describing the pool member's monitored state with a timestamp.
Conditions:
This may occur under the following conditions:
-- If an incremental config sync occurs from one high availability (HA) member to another member or to the device group:
+ The timestamp on monitor instances for all Nodes or Pool Members (as shown by 'tmsh show ltm monitor <type> <name>') may be updated on HA members receiving the incremental config sync.
+ If a Node or Pool Member has been marked DOWN by a monitor, its timestamp may be updated in the GUI (Node List/Pool-Member list) on HA members receiving the incremental config sync.
-- If a full/forced config sync occurs from one HA member to another member or to the device group:
+ The timestamp on monitor instances for all Nodes or Pool Members (as shown by 'tmsh show ltm monitor <type> <name>') may be updated on HA members receiving the incremental config sync.
+ The timestamp for all Nodes or Pool Members may be updated in the GUI (Node List/Pool-Member list) on HA members receiving the incremental config sync.
-- If a config load occurs:
+ The timestamp on monitor instances for all Nodes or Pool Members (as shown by 'tmsh show ltm monitor <type> <name>') may be updated on the HA member where the config load occurred.
+ The timestamp for all Nodes or Pool Members may be updated in the GUI (Node List/Pool-Member list) on the HA member where the config load occurred.
Impact:
The timestamp indicated next to the monitored-state description for an LTM Node or Pool Member indicates when the Node or Pool Member was updated in ways other than by its configured monitor. Thus, this timestamp may not indicate the actual time of the monitor event suggested by the description text.
Workaround:
None.
Component: TMOS
Symptoms:
During initialization, the SmartNIC driver reports a fixed MTU size. This prevents the MTU from being modified and results in the following error message in /var/log/tmm:
bigip1 notice xnet[00:08.0]: Error: Command 14 failed: 5
Conditions:
This error message is generated each time TMM is started or restarted.
Impact:
Jumbo frames are not supported by SmartNIC.
Workaround:
None.
Component: TMOS
Symptoms:
Warning log message generated to /var/log/ltm
warning mcpd[5494]: 01071859:4: Warning generated : cannot set MTU for eth1. No such device. For legacy VE driver.
Conditions:
Occurs when enabling or disabling VLAN-based Syn Cookies.
Impact:
No known functional impact. This is an incorrectly generated warning message.
Workaround:
None.
Component: TMOS
Symptoms:
imish crashes while checking the help strings of '(no) neighbor x.x.x.x fall-over bfd ?' when Border Gateway Protocol (BGP) is configured.
Conditions:
-- Configure BGP.
-- Check for help strings in imish using the '?' (question mark) character.
Impact:
imish crash.
Although imish crashes, BGP functionality is not impacted.
Workaround:
Avoid using '?' while entering the commands.
Component: Application Security Manager
Symptoms:
CSRF token expired message for AJAX calls is displayed incorrectly in alert message
Conditions:
This can occur if a BIG-IP administrator or user navigates to pages containing AJAX calls that modify a policy when the CSRF token is already expired. This occurs very rarely.
Impact:
Each letter in message is shown in new line in alert message
Component: Local Traffic Manager
Symptoms:
The iRules LX workspaces belong on only one device in a Device Service Cluster (DSC). If you have the same iRules LX workspace configured on multiple devices and then perform a config sync operation, the sdmd daemon cores.
Conditions:
-- Multiple devices configured with the same iRules LX workspace in a DSC.
-- Change one of the devices such that the configuration requires a config sync.
-- Perform the config sync.
Impact:
The sdmd daemon cores. Although having multiple devices configured with the same iRules LX workspace is an incorrect configuration, sdmd should not core.
Workaround:
When the iRules LX workspace is correctly configured, i.e., on only one device in a DSC, there is no need to config sync, so this issues does not occur.
Component: Advanced Firewall Manager
Symptoms:
All mitigation for IPv6 Duplicate Extension Headers DoS vector happens at the software level. The counter in int_drops column reads 0 (zero) in dos_stat tmctl for that vector.
Conditions:
Double IPv6 headers in the packets (except Destination Options header).
Impact:
You cannot offload DDoS vector 22 to FPGA hardware. Only the software-based DDoS mitigation is supported.
Workaround:
Use software DDoS mitigation for this vector.
Component: TMOS
Symptoms:
-- When a remotely authenticated user attempts login over the serial console, the username and password are accepted, but the session closes immediately thereafter.
-- Login over SSH is successful for the same user
Conditions:
-- Remote authentication (e.g., RADIUS, TACACS, LDAP) and role mapping configured on the BIG-IP system.
-- Attempted login over the serial console for a remotely authenticated user who has been assigned a role.
Impact:
Remotely authenticated users cannot log in over the serial console.
Workaround:
Using either of the following workaround:
-- Log in over SSH instead
-- If acceptable (taking into account security considerations), enable terminal access for all remote users regardless of assigned role, using 'tmsh modify auth remote-user remote-console-access tmsh' or within the GUI.
Component: Local Traffic Manager
Symptoms:
The GUI saves SSL certificate/CSR subject fields data into SSL.CertRequest.* DB variables to use them in pre-populating subject fields for subsequent modifications.
Conditions:
-- SSL certificate/CSR modification through GUI.
-- Changing the content of the SSL.CertRequest.* DB variables.
-- High availability (HA) configuration.
Impact:
HA devices go out of sync.
Workaround:
SSL.CertRequest.* DB variables are used only as GUI SSL certificate/CSR pre-populated suggestions.
You can still review and modify them before completing SSL certificate/CSR modification operation, so it is safe to sync them onto the high availability (HA) peer.
Component: Access Policy Manager
Symptoms:
Output from the 'show ltm virtual <vs> detail' command reports an incorrect value for the APM Module-Score.
Conditions:
-- Using GTM/DNS and APM.
-- Configure an access profile attached to a virtual server.
-- Configure a non-zero number for 'Max Concurrent Users' for the access profile.
-- Access the virtual server.
Impact:
GTM/DNS load balancing does not work as expected.
Workaround:
None.
Component: TMOS
Symptoms:
Guestagentd crashes on a vCMP guest.
Conditions:
This can occur during normal operation in a vCMP environment.
Impact:
Guestagentd crashes on the vCMP guest, and the vCMP host does not have accurate guest information, such as version, provisioning, high availability (HA) status, and tmm status.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
Ephemeral pool members may not be in the expected state if the corresponding FQDN template pool member is modified by one of several actions.
For example:
A. Ephemeral pool members may be missing from a pool in a partition other than Common, after reloading the configuration of that partition.
B. Ephemeral pool members may not inherit the 'session' state from the corresponding FQDN template pool member if the FQDN template pool member is disabled (session == user-disabled), the config is synced between high availability (HA) members, and BIG-IP is restarted.
Conditions:
Scenario A may occur when reloading the configuration of non-'Common' partition, e.g.:
-- tmsh -c "cd /testpartition; load sys config current-partition"
Scenario B may occur when an FQDN template pool member is disabled (session == user-disabled), the config is synced between HA members, and BIG-IP is restarted.
Impact:
Impacts may include:
- Missing ephemeral pool members, inability to pass traffic as expected.
- Ephemeral pool members becoming enabled and receiving traffic when expected to be disabled.
Workaround:
For scenario A, reload the entire configuration instead of just the individual partition.
For scenario B, it may be possible to work around this issue by checking the status of ephemeral pool members after BIG-IP restart, and toggling the 'session' value between user-enabled and user-disabled.
Component: Local Traffic Manager
Symptoms:
When the connection between the BIG-IP system and HSM (SafeNet) is interrupted, pkcs11d is unable to recover in some case.
Conditions:
Connection between the BIG-IP system and the HSM device is interrupted.
Impact:
SSL handshake failure.
Workaround:
Restart the pkcs11d process using the following command:
restart /sys service pkcs11d
Component: Access Policy Manager
Symptoms:
Active Directory queries are slow.
Conditions:
-- Active Directory (AD) authentication used
-- There are lots of AD caches in the environment, and users are in deeply nested groups.
Impact:
Active Directory query time can be excessive.
Component: Advanced Firewall Manager
Symptoms:
Bad destination detection threshold cannot accurately reflect the actual traffic pattern.
Conditions:
-- Enable bad destination and fully automatic mode.
-- Actual traffic is high.
Impact:
A small bad destination detection value is returned.
Workaround:
None.
Component: Global Traffic Manager (DNS)
Symptoms:
TMM fails to sign responses from BIND.
Conditions:
BIND has 'dnssec-enable no' in named.conf.
Impact:
TMM fails to sign responses from BIND.
Workaround:
Remove 'dnssec-enable no' from named.conf in options section.
Component: Access Policy Manager
Symptoms:
VPN Tunnel establishment fails with some ipv6 address
Conditions:
- APM is provisioned.
- Network Access with IPv6 virtual server is configured.
Impact:
VPN Tunnel cannot be established.
Workaround:
1. Disable the DB variable isession.ctrl.apm:
tmsh modify sys db isession.ctrl.apm value disable
2. Perform 'Apply Access Policy' for the access policy attached to the virtual server.
Important: The iSession control channel is needed if optimized apps are configured, so use this workaround only when 'No optimized apps are configured' is set (available in the GUI by navigating to Access :: Connectivity / VPN : Network Access (VPN) : Network Access Lists :: {NA resources} :: 'Optimization' tab).
Component: TMOS
Symptoms:
Being logged into the system in single user mode (emergency shell) causes a sudden automatic reboot after some time (~5-to-10 minutes, or longer).
Conditions:
-- Using iSeries platforms.
-- When logged into the emergency shell by appending rd.break to kernel command line.
Impact:
The device reboots after some time. Because of the automatic reboot, you cannot reliably use the emergency shell.
Workaround:
None.
Component: Access Policy Manager
Symptoms:
Error messages when navigating to Access :: Profiles/Policies : Customization : General Customization or Advanced Customization:
Button states incorrect on return from server after button push.
Conditions:
After creation of an Access Profile with Customization Type = modern (which is the default starting with v15.1.0).
Impact:
The GUI for General Customization or Advanced Customization of Access Profiles is not available.
Workaround:
Run the following commands, in sequence, from an SSH session:
cp --preserve=all /var/sam/www/client/customization-source/Common/modern/eps/default_eps.xml /var/sam/www/client/customization-source/Common/modern/eps/default_eps.xml.ORG
sed -e 's|b\>\ã|b\>\ã|' /var/sam/www/client/customization-source/Common/modern/eps/default_eps.xml.ORG > /var/sam/www/client/customization-source/Common/modern/eps/default_eps.xml
bigstart restart tomcat
Component: Application Security Manager
Symptoms:
ASM event logging stops working.
Conditions:
This can occur during normal ASM operation. It occurs after ASM executes 'No space in shmem' error disconnection mitigation, and this error is logged.
Impact:
ASM Policy Event Logging stop working; new event is not saved.
Workaround:
Restart asmlogd and pabnagd:
pkill asmlogd
pkill pabnagd
Component: Local Traffic Manager
Symptoms:
HTML profile performs a re-chunk even though HTML::disable has been executed in the HTTP_RESPONSE event.
Conditions:
Using HTML::disable in an HTTP_RESPONSE event.
Impact:
The HTML profile still performs a re-chunk.
Workaround:
None.
Component: TMOS
Symptoms:
The boot_marker is written to /var/log/btmp, but /var/log/btmp is a binary file.
A message similar to:
Apr 21 09:19:52 bigip1 warning sshd[10901]: pam_lastlog(sshd:session): corruption detected in /var/log/btmp
... may be logged to /var/log/secure.
Conditions:
-- Rebooting a BIG-IP.
Impact:
Since this file is unknowingly corrupt at first boot, any potential investigation needing this data may be compromised.
Workaround:
After bootup you can truncate the file.
$ truncate --size 0 /var/log/btmp
Component: TMOS
Symptoms:
Logging is not triggered for incomplete HTTP requests.
Conditions:
- HTTP profile is configured.
- Request-log profile is configured.
- HTTP request is incomplete.
Impact:
Logging is missing for incomplete HTTP requests.
Workaround:
None.
Component: TMOS
Symptoms:
The "tmsh show cm failover-status" command shows failover connection status "Error" on one device, but manual failover is working properly.
Conditions:
-- Two or more devices configured with high availability (HA) and are in sync.
-- The management IP address is changed on one of the devices
--show cm failover-status" on peer. You will see "show cm failover-status" returns "Error" status.
Impact:
The tmsh show cm failover-status command indicates an error, even though the devices are in sync and failover communication is working.
Workaround:
tmsh restart sys service sod
Component: Local Traffic Manager
Symptoms:
Loopback traffic (local traffic) destined to a virtual server might get dropped when the incoming packet matches a terminating connection flow.
Conditions:
-- VLAN filter is enabled on the virtual server created for loopback traffic processing.
-- An incoming packet matches a terminating connection flow (i.e., the connection flow terminates because of timeout, being dropped by iRule, etc.).
Impact:
Traffic that is matched against a terminating connection flow of a virtual is not processed by the virtual server.
Workaround:
Because this filter is ignored for loopback traffic, removing the 'Enabled On VLAN' filter at the virtual server mitigates the issue.
Component: TMOS
Symptoms:
AFM Debug Reroute feature does not work.
Conditions:
Attempting to use the AFM Debug Reroute Feature.
Impact:
Cannot use the AFM Debug Reroute feature.
Workaround:
None.
Component: Traffic Classification Engine
Symptoms:
There is an issue with the script used for the Traffic Intelligence (CEC (Classification Engine Core)) Hitless Upgrade that misses installing on some blades during install/deploy on VIPRION systems.
Symptoms include:
-- POST error in the GUI.
-- Automatic classification updates are downloaded successfully, but downloaded packages disappear after some time if you do not proceed to install/deploy.
Conditions:
-- CEC hitless update.
-- Using VIPRION chassis.
Impact:
Unable to auto-update Classification signature package on all slots, because the slot count reported for CEC is 0. These packages are installed only on the current slot.
Workaround:
Install the package manually on each slot.
Note: When you refresh the GUI page, the downloaded package appears in the 'Available to Install' list, and you can proceed to install on each slot.
Component: Service Provider
Symptoms:
With increasing traffic on a virtual server with an iRule configured to send TCP::respond, iRules operate fine until some threshold is reached, after which memory usage continually increases even though the traffic level remains stable. This memory growth increases until the reaper is activated and connections are removed.
Conditions:
A specific threshold of data is reached. The threshold varies, depending on the memory available on the BIG-IP system.
Impact:
Memory usage continually increases even though the traffic level remains stable. This memory growth increases until the reaper is activated and connections are removed. A tmm core is observed. Traffic disrupted while tmm restarts.
Workaround:
None.
Component: TMOS
Symptoms:
IPsec tunnels fails to remain established after initially working.
On the first ESP Security Associations (SAs) establishment, an IPsec tunnel works. After the expiry of the SAs causes a re-key, the keys are calculated incorrectly by the BIG-IP system. The BIG-IP system sends ESP packets to the remote peer, but the remote peer cannot decrypt the packet. Likewise, the BIG-IP system cannot decrypt packets from the remote peer.
This may also immediately present as a problem when trying to establish a second tunnel to the same peer.
Conditions:
- IPsec IKEv2 tunnel.
- A remote peer that is not another BIG-IP system.
- Elliptic curve groups (ECP) is used for Perfect Forward Secrecy (PFS).
Impact:
Multiple IPsec tunnels to the same remote peer cannot be established concurrently, or tunnels fail after a period of time.
Workaround:
Do not use ECP for PFS.
Component: WebAccelerator
Symptoms:
WAM image optimization can leak entity reference when demoting to unoptimized image.
WAM allows PNG files to be optimized to WEBP and JPG files to be optimized to JPEG XR formats, based on capabilities inferred from the client's User-Agent value. Once the optimized version is in the cache, internal check failures might cause the entity/document to be reverted to the unoptimized version. If this unoptimized version is already present in the cache, a reference to the corresponding entity is leaked, thus causing the entity to be held in memory along with attached resource/document objects and associated storage (UCI).
Conditions:
-- WAM-optimized PNG files (to WEBP) and JPG files (to JPEG XR) on tye system.
-- A policy change occurs that causes an internal check to fail.
Note: This can also occur in some cases without actual changes to the policy if the optimization step is skipped by wamd.
Impact:
WAM image optimization might leak entity reference.
Workaround:
None.
Component: TMOS
Symptoms:
The 'program' filter type does not work with the BIG-IP system's version of syslog-ng.
Conditions:
-- Using the 'program' expression in a syslog-ng filter.
Impact:
Unable to filter messages as expected.
Workaround:
None.
Component: TMOS
Symptoms:
When listing a property group, if you do not specify any specific properties within that group, the entire object is listed.
Conditions:
-- Using TMSH to list a property group of an object.
-- Not specifying any properties within the property group.
Impact:
Unexpected output.
Workaround:
None.
Component: Global Traffic Manager (DNS)
Symptoms:
Making changes to wide IP pools through GUI management do not take effect.
Conditions:
-- GTM configuration contains a sufficiently high number of pools (~ 15,000).
-- Using the GUI to assign a pool to a wide IP.
Impact:
Changes do not take effect. Unable to use the GUI to manage which pools are associated with a wide IP.
Workaround:
Use TMSH.
Component: TMOS
Symptoms:
When using the rewrite profile, unchunked server responses where content-type is not text/html or text/css also gets converted to chunked encoding in client-side. Also, the server response is missing a message-body (no content-type/content-length).
The client device receives 'Transfer-Encoding: chunked' in the message-header and receives a chunked body if the origin response has a message-body. The client receives a zero-length chunk if the origin response has no message-body.
Prior to BIG-IP version 15, chunking happens only if origin server response has content-type header set to either text/html or text/css.
Conditions:
- HTTP profile response chunking is set to 'sustain'.
- The virtual server has rewrite profile attached.
- Server response has content-type set not to text/html or text/css OR no content-type header.
Impact:
End users may notice a change in chunking behavior after upgrading from prior release.
Workaround:
Use response chunking mode 'unchunk'
Component: TMOS
Symptoms:
Configuration changes made in the BIG-IP GUI by a user with role 'Certificate Manager' do not result in the configuration being saved.
If the system is rebooted (or MCPD restarted) without saving the configuration, those changes will be lost.
Conditions:
-- User with role 'Certificate Manager'.
-- Changes made in GUI.
-- System rebooted.
Impact:
Loss of configuration changes.
Workaround:
Users with a 'Certificate Manager' role can save the configuration from tmsh:
tmsh save /sys config
Alternately, another user can save the configuration.
Component: Local Traffic Manager
Symptoms:
TMM crashes and generates a core file.
Conditions:
-- System is provisioned for at least ASM, AVR, and LTM.
-- An LTM pool is configured to use connection queuing.
-- The LTM pool is used on a virtual server with an analytics profile.
Impact:
Tmm might crash. Traffic disrupted while tmm restarts.
Workaround:
Disable connection queuing on the pool.
Component: Application Security Manager
Symptoms:
Attack signatures are marked as detected when they should be marked as excluded (i.e., a false positive).
Conditions:
-- A 100-continue transaction occurs in HTTP.
-- The internal parameter answer_100_continue is set to a non-default value of 0.
Impact:
False positive enforcement for header signature.
Workaround:
Set the answer_100_continue to 1 (default) on versions later than 15.0.0.
Component: Local Traffic Manager
Symptoms:
AFM virtual-server specific rules are being matched against control-plane traffic.
Conditions:
-- Broadcast OSPF configured.
-- AFM provisioned.
-- OSPF neighbor configured.
Impact:
OSPF neighborship is not formed.
Workaround:
Add an AFM route-domain policy.
Component: Local Traffic Manager
Symptoms:
SSL handshakes intermittently fail for virtual servers using HSM keys.
In /var/log/ltm you see errors:
err pkcs11d[6575]: 01680002:3: Key table lookup failed. error.
Conditions:
1. Keys were created on earlier versions of BIG-IP software with fipskey.nethsm wrapper, and the device was upgraded to 14.1.0 or later.
2. Keys were created on BIG-IP v14.1.0 or later directly, using fipskey.nethsm wrapper.
3. The platform is a multi-bladed Viprion.
This can occur after applying the workaround for ID758491:
https://cdn.f5.com/product/bugtracker/ID758491.html
Impact:
SSL handshakes that arrive on the secondary blade(s) fail.
Handshakes arriving on the primary blade work fine.
Workaround:
Re-install the Thales client after the upgrade.
Component: Local Traffic Manager
Symptoms:
The order of cipher results may change with no modification in the cipher group.
Conditions:
Click 'Update' in a cipher group in the GUI without making any changes.
Impact:
The order of the ciphers changes. During a handshake, SSL/TLS may not be able to select ciphers in the preferred order.
Workaround:
Create a cipher rule with the preferred cipher order and include only a single rule in cipher group allow list.
Component: TMOS
Symptoms:
When you try to update the frequency of live-update using iControl REST, it results in a java exception being returned instead of updating the value.
Conditions:
When a command for updating the frequency of live updates is executed using iControl REST in an ASM configured BIG-IP.
Impact:
You are unable to update the frequency of live-update via iControl REST.
Component: TMOS
Symptoms:
Traffic is not matching to the virtual server.
Conditions:
Using the GUI to configure traffic-matching-criteria by adding port-list to the virtual server.
Impact:
Traffic loss.
Workaround:
Creating traffic-matching-criteria from the command line
root@(localhost)(cfg-sync Standalone)(Active)(/Common)(tmos)# create ltm traffic-matching-criteria tmc_name_here destination-address-inline <IP ADDR>%10 route-domain <Route domain name>
Component: Application Security Manager
Symptoms:
XML Export does not work in configurations that have metacharacters or method overrides defined on URLs.
Conditions:
A policy that has metacharacter or method overrides defined on a URL is exported to XML format.
Impact:
Such a policy cannot be imported.
Workaround:
Use binary export/import or move/remove the problematic elements from the XML file:
* <mandatory_body>
* <operation_id>
Component: Application Visibility and Reporting
Symptoms:
AVR fails to send statistics to BIG-IQ. Lack of stats data in the BIG-IQ console.
Conditions:
-- The BIG-IP system is connected to the Data Collection Device (DCD), BIG-IQ.
-- DCD is restarted.
-- DCD does not send to BIG-IQ configuration instructions via REST interface due to ID 898341.
Impact:
Unable to collect statistics from BIG-IP system. Lack of stats data in the BIG-IQ console.
Workaround:
Restart avrd:
bigstart restart avrd
Component: Local Traffic Manager
Symptoms:
After reboot, no access to services host using fqdn nodes.
-- fqdn nodes are not populated with IP addresses.
-- Unable to access virtual servers served by pools using fqdn nodes.
Conditions:
The issue happens after the BIG-IP is rebooted.
-- when DNS server is accessed through a local virtual server.
-- Single arm cloud BIG-IP with virtual server listening for DNS requests to redirect.
Impact:
-- FQDN DNS requests bypassing the listening virtual server.
-- Unable to access the pools of those configured fqdn nodes.
Workaround:
-- restarting dynconfd.
-- Running a script to trigger off "Tmm ready" and either delete the bad flow(s) or a specific connflow entry.
-- change the dummy dns server to be something in the same subnet as the single interface.
Component: Local Traffic Manager
Symptoms:
If a TCP profile is configured with a syn-rto-base value that is lower than minimum-rto, the first retransmission might happen after syn-rto-base.
This behavior is encountered only if the BIG-IP system is unable to compute the new RTO value before the retransmission timer expires, meaning:
-- The BIG-IP system has not received a packet with a TCP timestamp reply.
-- The BIG-IP system has not received an ACK for a timed sequence number.
Conditions:
Configured value of syn-rto-base is lower than minimum-rto.
Impact:
Retransmission might happen sooner than expected.
Workaround:
There are two possible workarounds:
-- Avoid using a syn-rto-base value that is lower than the minimum-rto value (the default values are 3 seconds for syn-rto-base and 1 second for minimum-rto).
-- Consider enabling timestamps to allow faster RTT measurement.
Component: Local Traffic Manager
Symptoms:
Outgoing queries to backend dns server use incremented port numbers instead of being distributed random ports.
Conditions:
-- Fix of ID726176 is applied (see https://cdn.f5.com/product/bugtracker/ID726176.html )
Impact:
The port numbers are incremented.
Component: TMOS
Symptoms:
When merging a configuration that modifies the list of iRules a virtual server uses, you may encounter an error similar to:
01070621:3: Rule priorities for virtual server (/Common/virtual1) must be unique.
Conditions:
-- Merging a configuration using the TMSH 'replace' verb.
-- Replacing a virtual server's iRule in a way that adjusts priorities of the iRules.
Impact:
Unable to replace configuration using TMSH's 'replace' verb.
Workaround:
None.
Component: Access Policy Manager
Symptoms:
VMware has a restart desktop option to reboot the Horizon Agents, but APM does not support this feature on the webtop.
Conditions:
You wish to use the VMware Restart desktop feature for the Horizon Agents that are managed by the vCenter Server.
Impact:
Cannot restart the desktop (Horizon Agent) from the webtop by clicking the restart icon.
Workaround:
None.
Component: TMOS
Symptoms:
iControl REST async endpoint /mgmt/tm/task/util/ihealth behaving inconsistently:
-- A call to VALIDATE the async task is rejected with the error message: 'Operation is not allowed on component /util/ihealth.'
-- The task can be started by calling a different endpoint (e.g., /mgmt/tm/task/cli/script). In this case, the task completes immediately, however, a qkview generating iHealth util is still running. At the end, the qkview is generated.
Conditions:
-- Use iControl REST to create an async task for creating qkview using 'ihealth' with -n option (just generate file, do not upload to iHealth).
-- Try starting the async task by changing the status to VALIDATING.
Impact:
Patch for iControl REST endpoint is not successful. Patch operation is accepted by /mgmt/tm/task/cli/script/ but rejected by /mgmt/tm/task/util/ihealth.
Workaround:
None.
Component: TMOS
Symptoms:
An asynchronous task created on one endpoint can be started using some other endpoint
Conditions:
Create an asynchronous task e.g. creating qkview using ihealth
using endpoint /mgmt/tm/task/util/ihealth
Gather the task id of the created asynchronous task and send it to a different endpoint e.g. /mgmt/tm/task/cli/script
Impact:
The asynchronous task can be started using this endpoint but this is not intended behavior.
Component: TMOS
Symptoms:
When a blade fails (but not administratively disabled), other blades take 10 seconds (configured with db variable clusterd.peermembertimeout) to detect its absence. Until the blade failure is detected, egress traffic which used the failed blade's interfaces is dropped.
Conditions:
-- A multi-blade chassis.
-- Interfaces are trunked.
-- Some blades do not have directly attached interfaces.
-- A blade which does have directly attached interfaces fails.
Impact:
Some traffic is dropped until the failed blade is detected (10 seconds by default.)
Workaround:
Attach interfaces to all blades.
Component: TMOS
Symptoms:
Handling of duplicate internal connections can tear down and clean up the newest connection. Instead it should always remove the oldest.
Conditions:
When internal connections are re-established.
Impact:
The cleanup of previous connections may incorrectly tear down the new connection. Error messages are reported in the log when this happens, for example:
Duplicate connections between BCM56XXD1 and stpd7749-2. Closing the new one.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
Creating and deleting VLANs/self IPs might end up with a different number of GARP responses, depending on the BIG-IP software version.
You might notice the differences when comparing older and newer releases, for example, comparing v14.1.0 and earlier compared with versions older than v14.1.0.
Conditions:
This might become evident when you upgrade from an older version.
Impact:
There is no functional impact as a result of this discrepancy.
Workaround:
None.
Component: TMOS
Symptoms:
The devices in a high availability (HA) environment are out of sync in strange ways; config sync status indicates 'In Sync', but iApps such as SSL Orchestrator are out of sync.
Conditions:
-- high availability (HA) environment with two or more devices.
-- Gossip used for config sync. (Note: Gossip sync is used by BIG-IQ for BIG-IP config sync by iAppLX.)
-- A gossip conflict occurs for some reason.
You can detect gossip conflicts at the following iControl REST endpoint:
/mgmt/shared/gossip-conflicts
You can check gossip sync status at the following iControl REST endpoint:
/mgmt/shared/gossip
Impact:
If there are gossip conflicts, the devices requires manual intervention to get back in sync.
Workaround:
When two devices are out of sync with different generation numbers due to gossip conflict, you can use the following guidance to resolve the conflict:
1. Update devices info to use the same generation number.
2. This info found on REST Storage worker. Storage worker uses the selflink plus a generation number as the key to a given set of data.
3. Add the data from the unit with the highest generation number to the other unit.
4. Must also take care to increase the generation number on the new data to match that of the highest generation
Commands used:
1. Look for GENERATION_MISSING and gossip-conflict objects:
tmsh list mgmt shared gossip-conflicts
2. Get the 'selflink in remoteState' attribute. This self link is same across all devices and checks on the browser with each device to discover the device that is on the highest generation number:
tmsh list mgmt shared gossip-conflicts <OBJECT_ID>
3. Now you know what device contains the most recent version of your data, run this command to get up-to-date data:
restcurl /shared/storage?key=<everything after 'https://localhost/mgmt/' on selflink>
4. Make a post to the out-of-date device that includes the info from the up-to-date device as the post body:
restcurl -X POST /shared/storage -d '{<data from above command>}'
Component: TMOS
Symptoms:
Virtual server configured with:
-- Destination address in a non-default route-domain, for example:
0.0.0.0%100/0
-- The configuration uses a destination port list.
Conditions:
Modify the virtual server's port-list to a different one.
Impact:
Mcpd generates a core, and causes services to restart and failover.
Workaround:
None.
Component: Service Provider
Symptoms:
After modifying an MRF transport-config to use a different TCP profile, TMM must be restarted for this change to take effect. tmm crash
Conditions:
-- Using MRF with a transport-config.
-- Modifying the transport-config so that it uses a different TCP profile.
Impact:
Expected changes do not take effect until TMM is restarted.
Workaround:
Restart TMM.
Note: Traffic is disrupted while tmm restarts.
Component: Local Traffic Manager
Symptoms:
TCP analytics reports very large goodput values under rare conditions.
Conditions:
-- TCP or FastL4 filter is in use.
-- TCP analytics is enabled.
-- A specific sequence number space is observed during the data transfer.
Impact:
TCP reports a very large goodput value to AVR. This also impacts the average goodput reports as the high value reported shifts the average value to a considerably large one.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
As of BIG-IP version 14.1, HTTP iRule commands that inspect HTTP state after the commands HTTP::respond, HTTP::redirect, and HTTP::retry returns errors instead of returning corrupt data (https://support.f5.com/csp/article/K23237429).
When the NTLM profile is configured, it does the same through a built-in TCL rule where among several things, it tries to check if HTTP::cookie exists. If a profile like HTTP exists wherein a redirect/respond/retry is configured, it results in a TCL error informing the admin that they are accessing an invalid HTTP state.
Conditions:
-- NTLM profile is configured alongside HTTP profile
-- One of the redirect/respond/retry commands has been executed before the NTLM profile accesses the state of HTTP (for ex. HTTP::collect, HTTP::close, HTTP::cookie, etc.).
Impact:
Tcl error is seen in /var/log/ltm informing the admin that the iRule operation executed after HTTP::redirect/retry/respond is not supported.
For example -
TCL error: _plugin_rule_/Common/ntlm_default_iis <HTTP_REQUEST> - ERR_NOT_SUPPORTED (line 1) invoked from within "HTTP::cookie exists [PROFILE::ntlm insert_cookie_name]"
Component: Local Traffic Manager
Symptoms:
MCPD crash when modifying rewrite profile.
Conditions:
-- More than one rewrite profile is configured.
-- At least two rewrite profiles are referencing each other circularly.
Impact:
MCPD crash. For a Device Service Cluster this results in a failover. For a standalone system, this results in an outage.
Workaround:
Do not create circular references with profiles.
Component: Local Traffic Manager
Symptoms:
An error like the example below when defining "any" protocol after previously defining traffic-matching-criteria with explicit protocols.
01b90011:3: Virtual Server /Common/vs-tcp's Traffic Matching Criteria /Common/vs-tcp_IP_VS_TMC_OBJ illegally shares destination address, source address, service port, and ip-protocol with Virtual Server /Common/vs-any destination address, source address, service port.
Conditions:
-- Previously defining traffic-matching-criteria with explicit protocols
-- Afterwards defining virtual server with "any" protocol
Impact:
Cannot define a valid virtual server with "any" protocol
Workaround:
Component: Access Policy Manager
Symptoms:
The autodosd process crashes occasionally due to the division by zero.
Conditions:
It happens when the autodosd process receives zero value from tmm.
Impact:
Autodosd is rebooted.
Component: TMOS
Symptoms:
After the ISO upgrade, any attempt to access the SSL Orchestrator guided configuration user interface results in the following error:
The requested URL /iapps/f5-iappslx-ssl-orchestrator/sgc/sgcIndex.html was not found on this server.
Conditions:
Upgrade the BIG-IP system.
Impact:
Cannot perform SSL Orchestrator configuration tasks using the SSL Orchestrator guided configuration user interface.
Workaround:
(1) Query the f5-iappslx-ssl-orchestrator package ID (9beb912b-4f1c-3f95-94c3-eb1cbac4ab99), and use the returned ID in the following step.
restcurl shared/iapp/installed-packages | jq -r '.items[] | select(.appName=="f5-iappslx-ssl-orchestrator") | .id'
9beb912b-4f1c-3f95-94c3-eb1cbac4ab99
(2) Delete existing f5-iappslx-ssl-orchestrator package references.
restcurl -X DELETE shared/iapp/global-installed-packages/9beb912b-4f1c-3f95-94c3-eb1cbac4ab99
restcurl -X DELETE shared/iapp/installed-packages/9beb912b-4f1c-3f95-94c3-eb1cbac4ab99
(3) Stop the REST framework daemons.
bigstart stop restjavad restnoded
(4) Make sure the /var/iapps/www/ directory exists.
mkdir -p /var/iapps/www/
(5) Create the RPMS.save directory.
mkdir -p /var/config/rest/iapps/RPMS.save
(6) Check if the current f5-iapplx-ssl-orchestrator RPM (e.g., 14.1.0-5.5.8) is present at the default location.
ls -la /var/config/rest/iapps/RPMS/
-- If it is not present, try to get it from either the /usr/share/packages/f5-iappslx-ssl-orchestrator/ directory or from the remote high availability (HA) peer device (/var/config/rest/iapps/RPMS/). Make sure the RPM version matches the current SSL Orchestrator configuration/version (e.g., 14.1.0-5.5.8). If there is no RPM available (anywhere), you cannot continue with this workaround.
-- If it is present, copy the RPM to /var/config/rest/iapps/RPMS/ (locally).
(7) Copy the current f5-iapplx-ssl-orchestrator (e.g., 14.1.0-5.5.8) RPM to the RPMS.save directory.
cp /var/config/rest/iapps/RPMS/f5-iappslx-ssl-orchestrator-14.1.0-5.5.8.noarch.rpm /var/config/rest/iapps/RPMS.save/
(8) Make sure you have only one f5-iappslx-ssl-orchestrator RPM in the RPMS.save/ directory and that it matches the RPM version. Remove other RPMs, if any.
(9) Remove the current SSL Orchestrator user interface artifacts.
rm -rf /var/iapps/www/f5-iappslx-ssl-orchestrator/
rm -rf /var/config/rest/iapps/f5-iappslx-ssl-orchestrator
(10) Restart the REST framework.
bigstart restart restjavad restnoded
(11) Wait at least 30 seconds.
(12) Open TMUI (the GUI) on the affected device, and navigate to SSL Orchestrator :: Configuration.
(13) The SSL Orchestrator Self-Guided Configuration page should initialize and eventually load successfully.
Component: Global Traffic Manager (DNS)
Symptoms:
A virtual server which is actually down and should show red is reported as up and shows green.
Conditions:
This issue happens when a virtual server is marked down by the system due to inheriting the status of its parent link.
Note: This issue only affects Link Controller systems, and not DNS/GTM systems.
Impact:
The WebUI cannot be used to reliably assess the status of Wide IP members (virtual servers).
Workaround:
Use the tmsh utility in one of the following ways to inspect the status of Wide IP members:
# tmsh show gtm pool a members
# tmsh show gtm server virtual-servers
Component: Application Security Manager
Symptoms:
Incorrect filter applied when you are redirected from Charts page to Requests Log.
Conditions:
Select request status in filter in Charts page and apply filter and then click View Requests on the bottom of Charts page.
Impact:
Filter not applied in Requests Log page after redirect.
Workaround:
Filter can be applied manually after redirect.
There is mismatch between AVR request types and ASM request statuses, e.g., Blocked in AVR includes Unblocked as well.
Component: TMOS
Symptoms:
The tpm-status command incorrectly reports system integrity status as 'Invalid' even when system software is not modified.
Conditions:
-- BIG-IP software v14.1.0 or later version.
-- EHF installed on TPM-supported BIG-IP platform.
Impact:
Incorrect presentation of system software status.
Workaround:
None.
Component: TMOS
Symptoms:
When modifying the pool for a virtual server, address translation and port translation checkboxes are enabled irrespective of their initial state.
Conditions:
-- Creating a virtual server using the GUI
-- Advanced Configuration is selected
-- Address Translation or Port Translation checkboxes are initially unchecked
-- You modify a pool from this screen
Impact:
Virtual server is created with address and port translation enabled.
Workaround:
You can disable it by again editing the virtual server.
Component: TMOS
Symptoms:
You have BIG-IP Virtual Edition (VE) v13.1.x affected with ID 774445, and its workaround is in place.
echo "device driver vendor_dev 15ad:07b0 unic" >> /config/tmm_init.tcl
After upgrading to a newer version, interfaces are down.
Conditions:
-- Virtual Edition environment affected by the ID774445.
-- Apply the workaround described in in Final - K74921042: BIG-IP VE may fail to process traffic after upgrading the VMware ESXi 6.7 host to Update 2 (or later) :: https://support.f5.com/csp/article/K74921042.
Impact:
Interfaces are down.
Workaround:
1. Edit the /config/tmm_init.tcl file to remove the following line:
device driver vendor_dev 15ad:07b0 unic
2. Reboot into into the new software version.
3.Restart TMM:
tmsh restart sys service tmm
Component: TMOS
Symptoms:
The intended screen does not show when you navigate in the WebUI to either of the following locations:
-- System :: Certificate Management :: Device Certificate Management->Device Trust Certificates
-- DNS :: GSLB :: Servers :: Trusted Server Certificates
The system returns the following error:
An error has occurred while trying to process your request.
Additionally, a Java stack trace is also logged to the /var/log/tomcat/catalina.out file.
Conditions:
An extraneous SSL CSR file is present in the /config/big3d or /config/gtm directory.
-- When the extraneous file is in the /config/big3d directory, the System :: Certificate Management :: Device Certificate Management :: Device Trust Certificates screen is affected.
-- When the extraneous file is in the /config/gtm directory, the DNS :: GSLB :: Servers :: Trusted Server Certificates screen is affected.
Impact:
The WebUI cannot be used to inspect those particular SSL certificate stores.
Workaround:
The /config/big3d and /config/gtm directories are meant to contain only one file each (client.crt and server.crt, respectively).
You can resolve this issue by inspecting those directories and removing any file that may have been accidentally copied to them.
For more information on those directories, refer to: K15664: Overview of BIG-IP device certificates (11.x - 15.x) :: https://support.f5.com/csp/article/K15664.
Component: Local Traffic Manager
Symptoms:
When an Internal Virtual Server is created without an existing 0.0.0.0 virtual address, it will have the state "disabled-by-parent".
Conditions:
-- An Internal Virtual Server is created without an existing 0.0.0.0 virtual address.
Impact:
The Internal Virtual Server will be considered unavailable and will not process traffic.
Workaround:
Create a 0.0.0.0 virtual address prior to creating the Internal Virtual Server.
Component: Local Traffic Manager
Symptoms:
A wrong OCSP status entry in SessionDB is returned during a cache lookup due to using a wrong input parameter - certificate serial number. The result is wrong OCSP status is used in the SSL handshake.
Conditions:
If OCSP object is configured in a clientSSL or serverSSL profile.
Impact:
A wrong OCSP status may be reported in the SSL handshake.
Component: TMOS
Symptoms:
A 128-character limit for BWC policy object names is enforced and reports an error:
01070088:3: The requested object name <name> is invalid.
Conditions:
Attempting to create a BWC policy object with a name longer than 128 characters.
Impact:
Unable to create BWC policy objects with names that have more than 128 characters.
Workaround:
Use fewer than 128 characters when creating a BWC policy.
Component: Local Traffic Manager
Symptoms:
A policy rule based on SSL SNI at SSL client hello is not triggered for a TLS1.3 connection.
Conditions:
-- LTM policy rule specifying SSL client hello SNI.
-- TLS1.3 connection.
Impact:
Policy rule not triggered for TLS1.3.
Workaround:
None.
Component: Access Policy Manager
Symptoms:
RDP resource with a user-defined address cannot be launched from the webtop when configured with modern customization.
After requesting the RDP file for a remote address, the RDP file fails to download and the system reports an error message:
Logon failed. Connection to your resource failed. Please click the Try Again button to try again or Close button to close this dialog.
Conditions:
-- Webtop with modern customization.
-- RDP resource with a user-defined address is assigned to the webtop.
Impact:
Cannot use remote desktop resource with user-defined addresses.
Workaround:
As the problem is with modern access policy with modern webtop, a quick workaround:
1. Create a standard access policy with standard webtop (it is similar to modern access policy and modern webtop):
-- 1.1 GUI: Access :: Profiles / Policies :: Create :: {choose Customization Type as 'Standard').
-- 1.2 GUI: Access :: Webtops :: Create :: {choose Customization Type as 'Standard').
Recreate similar access policy as modern access policy that is showing this problem.
If manually re-creating similar standard access policy is not possible, there is no workaround.
Component: Access Policy Manager
Symptoms:
TMM leaks memory and eventually crashes when it cannot allocate any more memory.
Conditions:
OAuth agent is used in APM per-request policy subroutine and authentication fails.
Impact:
Over a period of time, TMM crashes, as it is unable to allocate any more memory. Traffic is disrupted while tmm restarts.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
When an HTTP request contains the 'Connection: close' header, the BIG-IP system shuts the TCP connection down. If a virtual server has a OneConnect profile configured, the BIG-IP system fails to close the connection for HEAD requests disregarding a client's demand.
Conditions:
-- A virtual server has HTTP and OneConnect profiles.
-- An HTTP request has the method HEAD and the header 'Connection: close'.
Impact:
Connection remains idle until it expires normally, consuming network resources.
Workaround:
None.
Component: TMOS
Symptoms:
During config sync, you see error messages in the logs:
save_master_key(master): Not ready to save yet.
Conditions:
UCS load or configuration synchronization that includes encrypted objects.
Impact:
Many errors seen in the logs.
Workaround:
None.
Component: TMOS
Symptoms:
Unable to confirm the supported encryption types.
enable or add BGP routing prorotol to a route domain
imish >> enable >> conf t >> router bgp 20065004 >> neighbor 1.2.3.4 password ?
b7000.lab[0](config-router)#neighbor 1.1.1.1 password ?
WORD Encryption Type or the password
Conditions:
Configuring the bgp neighbor with encryption password.
Impact:
Unable to confirm the supported encryption types.
Workaround:
None.
Component: Application Security Manager
Symptoms:
There is mismatch between server and GUI timezone treatment for Turkey/Istambul timezone.
Conditions:
User sets Turkey/Istambul timezone on BIG-IP
Impact:
When filtering logs by time period, results differ from set period by an hour
Workaround:
Define time period one hour earlier for filtering ASM logs
Component: Local Traffic Manager
Symptoms:
SYNs received with TSVal <= TS.Recent are dropped without sending an ACK in FIN-WAIT-2 state.
Conditions:
-- Timestamps are enabled in TCP profile.
-- Local TCP connection is in FIN-WAIT-2 state.
-- Remote TCP connection abandoned the flow.
-- A new TCP connection sends a SYN with TSVal <= TS.Recent to the local connection.
Impact:
The new TCP connection cannot infer the half-open state of Local TCP connections, which prevents faster recovery of half-open connections. The local TCP connection stays around for a longer time.
Workaround:
There are two workarounds:
-- Reduce the Fin Wait 2 timeout (the default: 300 sec) so that TCP connection is terminated sooner.
-- Disable TCP Timestamps.
Component: Global Traffic Manager (DNS)
Symptoms:
Stale /var/run/iqsyncer.pid file is causing a new iqsyncer application to exit immediately after start.
Conditions:
iqsyncer applications is killed by Linux kernel or any other reason causing a stale iqsyncer pid file
Impact:
Gtm config changes and gtm_add operations are blocked
Workaround:
Remove iqsyncer pid file manually or reboot
Component: Local Traffic Manager
Symptoms:
Traffic drop occurs.
Conditions:
Source MAC in the ARP header and the Ethernet header do not match.
Impact:
The BIG-IP system drops these packets.
Workaround:
None.
Component: Application Security Manager
Symptoms:
When removing not last selected values in Systems or Tags filter, last one is removed.
Conditions:
Select several values in Systems or Tags filter and then remove not the last value.
Impact:
Filter incorrectly displayed.
Workaround:
None.
Component: WebAccelerator
Symptoms:
BIG-IP WAM/AAM provides a faster cache store called small object cache. To get into this cache, an object must have its size below a threshold defined in BigDB variable wam.cache.smallobject.threshold. BIG-IP does not always pickup this value after a restart of TMM.
Conditions:
- WAM/AAM is provisioned;
- A virtual server is configured with a webacceleration profile having a web application.
Impact:
When small object cache has a non-default value, it may incorrectly place an object into Small Object cache (faster cache store) or MetaStor (slower cache store), causing performance impact.
Workaround:
Reset wam.cache.smallobject.threshold value.
Component: WebAccelerator
Symptoms:
BIG-IP system software allows you to cache HTTP responses with WAM/AMM web applications. There is a special storage location for small-size objects. If a caching object is about to exceed a threshold limit, the BIG-IP system might change its caching storage to MetaStor. A fix for ID 792045 introduced an issue for instances in which it does not, which resulted in not serving a cached object.
Conditions:
-- WAM/AAM is provisioned.
-- Virtual server has a webacceleration profile with a web application.
-- The BIG-IP software contains a fix for ID 792045.
Impact:
The BIG-IP system resets a connection with an error, a cached object is not served, and the rendering of a client's webpage is not correct.
Workaround:
None.
Component: Application Security Manager
Symptoms:
When a URL starts with double slashes (i.e. "http://HOST//path"), and Bot Defense Profile decides to perform simple redirect, the request results with loading failure.
Conditions:
-- Bot Defense profile on blocking mode (or "Verification and Device-ID Challenges in Transparent Mode" is enabled) is attached to a virtual server.
-- A request is sent to a URL starting with double slash, to a non-qualified URL, during the profile's grace period.
Impact:
Request is not loaded (failure message is seen on browser), and the browser may be identified as a suspicious browser by Bot Defense.
Workaround:
None.
Component: Global Traffic Manager (DNS)
Symptoms:
Upon close inspection of the statistics of a particular DNS Cache, for example by running the command 'tmsh show ltm dns cache resolver <name>', you realize that the 'Total Responses' counter for the cache is not incrementing as much as it should be.
Specifically, by comparing the counter with packet captures or the stats of the DNS Profile, you realize the system is under-reporting 'Total Responses'.
Conditions:
The virtual server using the DNS Cache also uses an iRule which happens to include a suspending command (e.g., 'table') under the DNS_RESPONSE event.
Impact:
The incorrect DNS Cache statistics may confuse or mislead a BIG-IP Administrator.
No traffic impact exists as part of this issue. Responses are still being served from the cache even when the counter says they are not.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
Large POST requests are getting occasionally reset and you see the following in /var/log/ltm:
err tmm[19279]: 011f0007:3: http_process_state_cx_wait - Invalid action:0x100011 clientside
Conditions:
-- An HTTP iRule is configured on a virtual server
-- A large POST request arrives on the virtual server
Impact:
Possible connection failure.
Component: TMOS
Symptoms:
A user is unable to log in using remote LDAP.
Conditions:
-- BIG-IP systems are configured to use LDAP authentication.
-- Remote user has no search permissions on directory
Impact:
Authentication does not work.
Workaround:
Grant search permissions to the user in LDAP.
Component: Local Traffic Manager
Symptoms:
The following messages are found in the QKViews:
"bigipA notice MCP bulk connection aborted, retrying"
"bigipA notice Initiating TMM shutdown"
Prior to this, the TMM process logs that it is waiting for its instances to reach different states. For example,
"localhost notice ixlv(1.3)[0:7.0]: Waiting for tmm1 to reach state 1..."
In the /var/log/ltm file, the following message are found sometimes.
"bigip1 crit tmm9[19358]: 01230017:2: Unable to attach to PCI device 00:09.00 for Interface 1.5"
Conditions:
BIG-IP VE with SR-IOV enabled on a Red Hat Enterprise Linux 7.7 which is a part of Red Hat OpenStack Platform 13
Impact:
The TMM process restarts without a core file repeatedly.
Traffic disrupted while tmm restarts.
Component: TMOS
Symptoms:
A General Database Error message is shown when you click the Instances tab of a certificate bundle / certificate / Key listed under the System :: Certificate Management : Traffic Certificate Management : SSL Certificate List.
Conditions:
-- The selected SSL Certificate does not have a certificate or key listed under it.
-- You click the instances tab of the properties page of the SSL Certificate.
Impact:
GUI shows an error screen.
Workaround:
Avoid clicking the instance tab if there is no key or certificate associated with the SSL Certificate / Bundle.
Component: TMOS
Symptoms:
After upgrading, CGNAT is de-provisioned and tmm is restarted after config load.
Conditions:
- CGNAT provisioned prior to upgrade
- Upgrade from 13.1.0 to 15.1.0.1 and reboot
Impact:
-- CGNAT is de-provisioned
-- Tmm restarts
Workaround:
After upgrading, re-provision CGNAT:
tmsh modify sys provision cgnat level <level>
tmsh save sys config
Component: Carrier-Grade NAT
Symptoms:
There is a difference in active port block counter between statistics collected in TMM and actual allocations in 'lsndb list pba'.
Conditions:
The issue happens when the port block allocation process fails after incrementing the active port blocks counter.
Impact:
No functional impact. But the stats counters will be incorrect.
Component: Local Traffic Manager
Symptoms:
Tmm is running at 100% CPU even under light network load. The 'tmctl tmm/ndal_tx_stats' command shows a high number of packet drops. The 'tmctl tmm/ndal_tx_stats' indicates a large number of queue full events.
Conditions:
-- BIG-IP Virtual Edition.
-- There are underlying network performance issues causing the transmit queue to be full (e.g., a non-SR-IOV virtual machine environment).
-- Upgrading from BIG-IP v12.x to BIG-IP v14.x.
Impact:
NDAL's busy polling runs the tmm CPU usage to 100%.
Workaround:
Correct the underlying networking/virtualization issue.
Component: TMOS
Symptoms:
After a long uptime interval (i.e., the sod process has been running uninterrupted for a long time), HA Group failover may not complete despite an HA Group score change occurring. As a result, a BIG-IP unit with a lower HA Group score may remain as the Active device.
Note: Uptime required to encounter this issue is dependent on the number of traffic groups: the more traffic groups, the shorter the uptime, e.g.:
-- 1 floating traffic group: 2485~ days.
-- 2 floating traffic groups: 1242~ days.
-- 4 floating traffic groups: 621~ days.
-- 8 floating traffic groups: 310~ days.
-- 9 floating traffic groups: 276~ days.
Note: You can confirm sod process uptime in tmsh:
# tmsh show /sys service sod
Conditions:
HA Group failover configured.
Note: No other failover configuration is affected except for HA Group failover, specifically, these are not affected:
o VLAN failsafe failover.
o Gateway failsafe failover.
o Failover triggered by loss of network failover heartbeat packets.
o Failover caused by system failsafe (i.e., the tmm process was terminated on the Active unit).
Impact:
HA Group Active/Standby state transition may not complete despite HA Group score change.
Workaround:
There is no workaround.
The only option is to reboot all BIG-IP units in the device group on a regular interval. The interval is directly dependent on the number of traffic groups.
Component: Application Security Manager
Symptoms:
An attack signature is not detected.
Conditions:
-- The payload is filled with the percent character in between every other character.
-- The bad unescape violation is turned off.
-- The illegal metacharacter violation is turned off.
Impact:
An attack goes undetected.
Workaround:
Turn on the bad unescape violation or the metacharacter violation.
Component: Access Policy Manager
Symptoms:
The entityID property of SAML Service Provider (SP) object ('apm aaa saml') accepts only a valid URI as the value if host is empty. All other values are deemed invalid.
This creates a less than optimal configuration experience in certain use-cases. For instance, when the deployment contains two SAML SP configuration objects that are essentially identical, with the only difference being the entityID value, validation prevents reusing the same object, and mandates creation of two independent configuration objects.
Conditions:
-- The BIG-IP system is used as a SAML SP with two or more SP configuration objects.
-- The only difference between two (or more) configured SP configuration objects is the value of entityID.
Impact:
None. This is a usability enhancement.
Workaround:
Creating multiple SP objects.
Component: TMOS
Symptoms:
When a saved UCS is attempted to be restored in a new BIG-IP Virtual Edition (VE) in order to migrate the configuration, it fails.
load_config_files[28221]: "/usr/bin/tmsh -n -g -a load sys config partitions all " - failed. -- 01071412:3: Cannot delete IP (x.x.x.x) because it is used by the system config-sync setting.
Conditions:
The UCS load step might fail if the DB variable Provision.1NicAutoconfig is set to disable.
Impact:
The UCS restore fails.
Workaround:
The DB variable can be set to enable before loading the UCS.
# tmsh modify sys db provision.1nicautoconfig value enable
Component: Global Traffic Manager (DNS)
Symptoms:
“RESOLVER::name_lookup” returns null.
Conditions:
Response is larger than 512 bytes.
Impact:
“RESOLVER::name_lookup” returns empty answer.
Component: Global Traffic Manager (DNS)
Symptoms:
TMM Cored with SIGSEGV.
Conditions:
Impact:
Traffic disrupted while tmm restarts.
Component: Application Security Manager
Symptoms:
Under Session Hijacking :: Device Session Hijacking by Device ID Tracking, the note text below the 'enable' checkbox is shown in bold red color
Note : Device-ID mode must be configured in bot profile for this option to work.
Conditions:
This always occurs.
Impact:
The Note does not indicate a hazardous situation (as might be implied by the color), so the text should be black instead of red.
Workaround:
None.
Component: Application Security Manager
Symptoms:
A policy add/modify/delete fails with the following error:
-- crit g_server_rpc_handler_async.pl[19406]: 01310027:2: ASM subsystem error (asm_config_server.pl ,F5::ASMConfig::Handler::log_error_and_rollback): Failed on insert to DCC.VS_RAMCACHE (DBD::mysql::db do failed: Duplicate entry '375946375' for key 'PRIMARY').
Conditions:
This can occur when adding a policy. The chance of it occurring increases when there are many virtual servers.
Impact:
Every config update fails.
Workaround:
Figure out which virtual servers have the CRC collision (by looking into DCC.RAMCACHE_VS). Change the name of one of these virtual servers.
You can get the name of the affected virtual server by using the entry reported in the 'Duplicate entry' log, and running this command.
mysql -u root -p$(perl -MPassCrypt -nle 'print PassCrypt::decrypt_password($_)' /var/db/mysqlpw) -e 'SELECT * FROM DCC.VS_RAMCACHE WHERE vs_name_crc = 375946375'
Component: Traffic Classification Engine
Symptoms:
TMM crashes after updating the urldb blacklist.
Conditions:
-- The BIG-IP system is configured with URL blacklists.
-- Multiple database files are used.
Impact:
TMM restarts. Traffic disrupted while tmm restarts.
Workaround:
None.
Component: TMOS
Symptoms:
Script fails with:
stat: cannot stat '/shared/core/*.core.*': No such file or directory.
In addition, the system reports a message in /var/log/user and /var/log/messages when there are no core files:
Deleting file /shared/core/*.core.*
Conditions:
Coreexpiration script is run.
Impact:
No core is produced. In addition, there is no core deleted.
Workaround:
To resolve the issue, add the following line to the script:
for filename in /shared/core/*.core.*; do
+ [ -e "$filename" ] || continue
# Time of last modification as seconds since Epoch
Component: Application Security Manager
Symptoms:
When booting to a boot location for the first time, the system does not come on-line.
Conditions:
-- There is a large ASM configuration.
-- VLAN failsafe is configured, and the failsafe-action is something other than failover.
-- The BIG-IP system is an appliance.
Impact:
BIG-IP processes continually restart (vlan failsafe-action failover-restart-tm) or the BIG-IP system continually reboots (vlan failsafe-action reboot)
Workaround:
Either disable VLAN failsafe or set the failsafe-action to failover during an upgrade.
Component: Application Security Manager
Symptoms:
The JSON schema validation file creation from swagger's schema entry fails and errors are logged to /ts/log/asm_config_server.log.
Conditions:
-- API Protection used in a security policy
-- Swaggers's schema entry contains one or more fields that are incompatible between draft-04 and draft-07 of the JSON schema validation spec (e.g.: "exclusiveMinimum")
Impact:
Since BIG-IP is locked to draft-07, a security policy created from swagger file will not include some entities.
Workaround:
No workaround
Component: TMOS
Symptoms:
SessionDB messages sent from Active to Standby are dropped due to inconsistencies detected in the message. You see logs in /var/log/ltm:
SessionDB ERROR: received invalid or corrupt HA message; dropped message.
Conditions:
-- High availability (HA) pair configuration.
-- SessionDB messages sent from Active to Standby.
Impact:
Standby drops these messages
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
When a session variable key length is 65 KB, session mirroring fails for that specific key.
Conditions:
-- APM high availability (HA) setup.
-- Access Policy is configured and synced across both devices.
-- A session variable key of ~65 KB arrives
Impact:
The session key does not get mirrored to standby.
Workaround:
Component: Advanced Firewall Manager
Symptoms:
The dynamic white/black daemon (dwbld) daemon shows very large memory consumption on adding addresses to shun-list.
Conditions:
-- Adding a large number of IP addresses to the shun-list.
-- Viewing dwbl memory usage using:
config # top -p $(pidof dwbld)
Impact:
The dwbl daemon uses a large amount of memory. If memory goes to exhaustion, enforcement of dwbl does not occur.
Workaround:
None.
Component: Application Security Manager
Symptoms:
The Bot Defense profile adds P3P headers to every response when a cookie is set, even if the client browser is something other than Microsoft Internet Explorer.
Conditions:
Bot Defense Profile is attached to a virtual server.
Impact:
Deprecated P3P header is inserted in all responses, even though it is only required for Internet Explorer.
Workaround:
The value of the P3P header is globally configurable in the DB variable dosl7.p3p_header.
It is also possible to set the value to '<null>' and thus prevent the P3P header from appearing, but this may cause legitimate Internet Explorer browsers to be be blocked from accessing the web application.
Component: Access Policy Manager
Symptoms:
APM has several types of access policies for different deployment types, such as general per-request policies, OAuth policies, full webtop portal policies, and so on. One type of policy is designed for API clients, called API Protection.
API Protection requests are generally authenticated by user information present in an HTTP authorization header. APM then uses this authorization header data to authenticate users against an AAA server.
In addition to authentication, some deployments of API Protection also require authorization decisions to be performed against out-of-band data from external servers, typically group membership data from an external HTTP or LDAP server.
Conditions:
Administrators attempt to use HTTP Connector or LDAP Query in an API Protection type access policy.
Impact:
Administrators are not able to use HTTP Connector or LDAP Query in API Protection policies.
Component: TMOS
Symptoms:
After upgrading, the system encounters numerous issues:
-- Memory exhaustion (RAM plus swap) with no particular process consuming excessive memory.
-- High CPU usage with most cycles going to I/O wait.
-- System is unresponsive, difficult to log in, slow to accept commands.
-- Provisioning is incomplete; there is a small amount of memory amount assigned to 'host' category.
Conditions:
-- The configuration works in the previous release, but does not work properly in the release you are upgrading to.
-- Device is upgraded and the configuration is rolled forward.
-- There may be other conditions preventing the configuration from loading successfully after an upgrade.
Exact conditions that trigger this issue are unknown. In the environment in which it occurred, a datagroup had been deleted, but an iRule was still referencing it, see https://cdn.f5.com/product/bugtracker/ID688629.html
Impact:
-- System down, too busy to process traffic
-- Difficulty logging in over SSH might require serial console access.
Workaround:
Reboot to an unaffected, pre-upgrade volume.
-- If the system is responsive enough, use 'tmsh reboot volume <N>' on BIG-IP Virtual Edition (VE) or switchboot to select an unaffected volume.
-- If the system is completely unresponsive, physically powercycle a physical appliance or reboot a VE from an applicable management panel, then select an unaffected volume from the GRUB menu manually.
Note: This requires that you have console access, or even physical access to the BIG-IP device if you are unable to SSH in to the unit. On a physical device, a non-responsive system might require that you flip the power switch. For more information, see K9296: Changing the default boot image location on VIPRION platforms :: https://support.f5.com/csp/article/K9296, K5658: Overview of the switchboot utility :: https://support.f5.com/csp/article/K5658, and K10452: Overview of the GRUB 0.97 configuration file :: https://support.f5.com/csp/article/K10452.
Component: Policy Enforcement Manager
Symptoms:
When there is a failover event, there is a chance that some sessions will not move over to new active blade.
Conditions:
-- High availability (HA) environment.
-- A CMP state change occurs.
Impact:
For certain IP addresses that have failed to move to the new active device, a new session create request does not create/replace the current session because it is in inconsistent state.
Workaround:
None.
Component: TMOS
Symptoms:
Connections stall when dynamic BWC policy is changed via GUI and TMSH.
Conditions:
Issue is seen when you have a dynamic bandwidth control policy configured, and you make a change to the policy via the GUI and TMSH.
Impact:
Connection does not transfer data.
Workaround:
Restart TMM. Delete the relevant configuration, create a new configuration, and apply it.
Component: TMOS
Symptoms:
A tmm thread might stall while yielding the CPU, and trigger a failsafe restart of the tmm process.
Conditions:
Unknown
Impact:
Traffic disrupted while tmm restarts.
Workaround:
None known.
Component: Global Traffic Manager (DNS)
Symptoms:
The 'Reconnect' and 'Reconnect All' buttons (introduced in BIG-IP version 14.1.0 to restart some or all iQuery connections) do not work when clicked.
The 'Reconnect' button does not become enabled when a server is selected from the list, and an error is logged in the browser console.
The 'Reconnect All' button is clickable but returns the error "No response action specified by the request" when clicked.
Conditions:
You have accessed the buttons via the following WebUI path:
DNS > GSLB > Data Centers > [dc name] > Servers
Impact:
The buttons do not work, making the corresponding feature unavailable from the WebUI.
Workaround:
Access the buttons via the following alternative WebUI path:
DNS > GSLB > Servers
Component: Local Traffic Manager
Symptoms:
Multi-NIC instances fail to come up while using memory-mapped virtio device.
Running the command 'lspci -s <pci-id> -vv' results in the 'region' field reporting 'Memory at xxxxx'.
Conditions:
TMM crashes as soon as the BIG-IP system tries to come up.
Impact:
The BIG-IP system fails to attach to the underlying virtio devices.
Workaround:
Switch to the sock driver by overriding tmm_init.tcl.
For instructions on how to enable the sock driver, see the workaround in K74921042: BIG-IP VE may fail to process traffic after upgrading the VMware ESXi 6.7 host to Update 2 (or later), available at https://support.f5.com/csp/article/K74921042.
Component: Global Traffic Manager (DNS)
Symptoms:
iQuery incorrectly interprets iQuery SSL certificate times when they are using GenericTime instead of UTCTime.
Conditions:
An iQuery certificate using GenericTime instead of UTCTime.
Note that this would only occur with a date beyond the year 2049.
Impact:
Internal years are interpreted to be much later than they should be.
Workaround:
Use SSL certificates with UTCTime instead of GenericTime.
Component: Application Security Manager
Symptoms:
Clicking the 'Fix Automatically' button in the PCI Compliance page does not replace the insecure client SSL profile attached on an HTTP/2 virtual server, with a secure one. The compliance state shows as a red cross mark, indicating the virtual server to be noncompliant.
Conditions:
-- Clicking the 'Fix Automatically' button on the PCI compliance page.
-- A noncompliant PCI profile is attached to the HTTP/2 virtual server.
-- A PCI-compliant, client SSL profile with renegotiation disabled is available in the SSL profiles.
Impact:
The provision for enhanced configuring does not function as expected for HTTP/2-based virtual servers.
Workaround:
Manually configure a PCI-compliant profile in SSL profiles, with renegotiation disabled, and attach it to the virtual server.
Component: Application Security Manager
Symptoms:
For an HTTP/2 virtual server with an insecure client SSL profile attached, clicking the 'Fix Automatically' button on the PCI Compliance page creates a PCI-compliant client SSL profile, but fails to attach to the virtual server. The compliance state shows as a red cross mark, indicating the virtual server to be noncompliant.
Conditions:
-- No compliant PCI profile is attached to the HTTP/2 virtual server.
-- Click the 'Fix Automatically' button on the PCI Compliance page.
Impact:
The provision for enhanced configuring does not function as expected for HTTP/2-based virtual servers.
Workaround:
Manually configure a PCI-compliant profile in SSL profiles, with renegotiation disabled, and attach it to the virtual server.
Component: Local Traffic Manager
Symptoms:
iRules that execute a lot can make stats counters large enough to overflow in a relatively short amount of time (e.g., a couple of months).
Conditions:
Execute an iRule a lot (e.g., make the total number of executions greater than 32 bits) and check its stats.
Impact:
After the total number exceeds 32 bits, the counter stats are no longer valid.
Workaround:
None.
Component: Global Traffic Manager (DNS)
Symptoms:
Err (error) level messages in /var/log/gtm log when DNS (GTM) SSL monitors such as https are used and are unable to connect to the monitored target IP address:
err big3d[4658]: 01330014:3: CSSLSocket:: Unable to get the session.
These messages do not indicate the IP address or port of the target that failed to connect, and this ambiguity may cause concern.
Conditions:
-- SSL-based DNS (GTM) monitor assigned to a target, for example https
-- TCP fails to connect due to a layer 2-4 issue, for example:
- No route to host.
- Received a TCP RST.
- TCP handshake timeout.
Impact:
The system reports unnecessary messages; the fact that the monitor failed is already detailed by the pool/virtual status change message, and the target changing to a red/down status.
These messages can be safely ignored.
Workaround:
If you want to suppress these messages, you can configure a syslog filter.
For more information, see K16932: Configuring the BIG-IP system to suppress sending SSL access and request messages to remote syslog servers :: https://support.f5.com/csp/article/K16932.
Component: TMOS
Symptoms:
After rebooting the standby BIG-IP device, IKE SA's are not mirrored.
Conditions:
-- IPSEC is configured in a high availability (HA) environment
-- Standby device is rebooted
Impact:
IKE_SA's will have to be renegotiated.
The performance impact is minimal.
Component: TMOS
Symptoms:
The IKEv1 IPsec daemon racoon goes into an endless restart loop.
2020-01-02 08:36:36: ERROR: /etc/racoon/racoon.conf.BIG-IP:376: "}" duplicated sainfo: loc='ANONYMOUS', rmt='10.42.80.0/24', peer='ANY', id=0
2020-01-02 08:36:36: ERROR: fatal parse failure (1 errors)
2020-01-02 08:36:36: ERROR: failed to parse configuration file.
Conditions:
Duplicate wildcard traffic-selectors, one with ::/0 and one with 0.0.0.0/0, attached to different IPsec policies.
Impact:
IPsec IKEv1 tunnels cannot be established.
Workaround:
Configure duplicate traffic-selectors only when they are attached to interface mode IPsec policies.
Component: TMOS
Symptoms:
The vCMP CPU usage stats are incorrect when process on a secondary blade has the same PID as that of primary blade's qemu process.
Conditions:
A process on a secondary blade has the same PID as that of primary blade's qemu process.
Impact:
The vCMP CPU usage stats are intermittently incorrect.
Workaround:
None.
Component: Application Security Manager
Symptoms:
When pressing 'Create' button in
Security ›› Application Security : URLs : Allowed URLs : Allowed HTTP URLs page, the requested page is not loaded.
Conditions:
Policy with about 5000 and more parameters causes long loading time, which results in loading failure.
Impact:
The requested page (New Allowed HTTP URL...) is not loaded.
Workaround:
Use fewer parameters (less than 5000) per policy.
Component: TMOS
Symptoms:
Frequent config syncs and spamming of logs are occurring on BIG-IP devices in a high availability (HA) configuration.
Conditions:
Datasync CAPTCHA table is re-generated while CAPTCHA is being consumed by users.
Impact:
Sync to the datasync groups cause the sync status of the devices to fluctuate.
Component: Application Security Manager
Symptoms:
When a trying to install a new bot defense signature, the installation fails with the following log message:
com.f5.liveupdate.update.dosbotsignatures.file.Update.applyChanges.pl|INFO|Feb 10 13:22:12.924|7347|F5::Dos::BotSignatures::load_from_xml,,Cannot send updated objects to mcp: 01070265:3: The Bot Defense Signature (/Common/Headless Chromium, Chrome) cannot be deleted because it is in use by a Bot Defense Profile Signature Staging.
Conditions:
-- A Bot Defense Profile has a staged signature.
-- The staged signature points to something that does not exist in the update file.
Impact:
The new file cannot be installed.
Workaround:
Enforce the staged signature.
Component: Access Policy Manager
Symptoms:
When ACCESS::session irule is used in HTTP_RESPONSE event, the APM session creation fails with the following log in /var/log/ltm
No HTTP data available - command unsupported in event (line XX)session creation failed - Operation not supported (line XX)
Conditions:
Using ACCESS::session create command under HTTP_RESPONSE.
Impact:
Cannot create APM session using the ACCESS::session irule command.
Workaround:
The same irule ACCESS::session can be used under HTTP_REQUEST to create the APM session.
Component: TMOS
Symptoms:
Mcpd cores during config sync.
Conditions:
This has only been observed once. The device was going from standby to active, and the connection between the BIG-IP peers stalled out.
Impact:
Mcpd cores. Traffic disrupted while mcpd restarts.
Workaround:
Component: Local Traffic Manager
Symptoms:
-- RRD graphs are not updated.
-- System statistics are stale.
-- Commands such as 'tmsh show sys memory' may not complete.
-- qkview does not complete, as it runs "tmsh show sys memory'.
You may see errors:
-- err statsd[5005]: 011b0600:3: Error ''/var/rrd/endpisession' is too small (should be 15923224 bytes)' during rrd_update for rrd file '/var/rrd/endpisession'.
-- err statsd[5005]: 011b0600:3: Error '-1' during rrd_update for rrd file '/var/rrd/endpisession'.
Conditions:
Truncation of a binary file in /var/rrd.
Impact:
Stats are no longer collected. Statsd and rrdshim deadlock.
Workaround:
Remove the truncated file and restart statsd:
bigstart restart statsd
Component: TMOS
Symptoms:
After upgrading BIG-IP software, zrd fails to start.
There are errors in /var/log/daemon.log:
err named[19356]: open: /config/named.conf: permission denied
Conditions:
This can occur after upgrading, for example, when upgrading from version 14.1.0.6 to 15.0.1.1.
Impact:
DNS service disrupted as zrd fails to start after reboot.
Workaround:
Run the following command after upgrading:
restorecon -rF /var/named/
Component: TMOS
Symptoms:
Disabling DHCP on the management port causes sflow_agent to crash.
Conditions:
This does not always occur, but when it does occur, it crashes when disabling DHCP on the management port.
Impact:
sflow_agent crashes.
Workaround:
Do not disable DHCP on the management port
Component: Application Security Manager
Symptoms:
Applied Blocking Masks discrepancy between local/remote event log, ASM logging event logs both locally and remotely to BIG-IQ has discrepancy.
Conditions:
This occurs when "Applied Blocking Masks" logs are emitted on a device where local and remove event logging is configured.
Impact:
This is cosmetic but can lead to confusion.
Component: Local Traffic Manager
Symptoms:
When using two BIG-IP systems to mirror traffic, mirroring functions correctly if the default gateway VLAN names match; however, if default gateway VLAN names don't match, then the BIG-IP system does not mirror client-side packets to the peer, which causes the standby BIG-IP system to reset all client-side flows on failover.
Conditions:
-- Two BIG-IP LTM BIG-IP Virtual Edition (VE) systems configured as a high availability (HA) pair.
-- Default gateway VLAN names don't match between them.
Impact:
BIG-IP system does not mirror client-side packets to the peer, which causes the next-active device to reset all client-side flows on failover.
Upon failover all flows are being RST just like a typical failover scenario without mirroring implemented.
Workaround:
Use same VLAN name on all external VLANs that might be used for mirroring.
Component: Local Traffic Manager
Symptoms:
In this release, traffic does not pass on tagged VLANs when a BIG-IP Virtual Edition (VE) is deployed on a Hyper-V hypervisor.
This may manifest as traffic failing after an upgrade from earlier (unaffected) software versions.
Note: This functionality worked as expected in v13.x and earlier, and if the same VE is downgraded to v13.x, VLAN tagging functionality is restored.
This is due to an interoperability issue between RedHat Enterprise Linux (RHEL) and Microsoft Hyper-V, which seems to affect RHEL v7.3 and RHEL v7.5.
Hyper-V on Windows Server 2016 and Windows Server 2012 do not seem to identify the version of the built-in LIS correctly on Centos 7.3 or Centos 7.5 (which are built on RHEL 7.3 and RHEL 7.5 respectively).
Although there is a statement of support by Microsoft for VLAN tagging on RHEL 7.3 and 7.5 when running on Hyper-V, that functionality does not appear to work at present: Supported CentOS and Red Hat Enterprise Linux virtual machines on Hyper-V :: https://docs.microsoft.com/en-us/windows-server/virtualization/hyper-v/Supported-CentOS-and-Red-Hat-Enterprise-Linux-virtual-machines-on-Hyper-V.
Conditions:
-- BIG-IP VE is deployed on a Hyper-V hypervisor.
-- VLAN configured in BIG-IP VE with tagged interfaces, e.g.:
net vlan external {
interfaces {
1.1 {
tagged
tag 4000
-- At present, VLAN tagging on the v14.x and v15.x releases does not work because those releases are running on CentOS 7.3 and 7.5 respectively, which both are affected by the MS/RHEL interoperability issue.
-- BIG-IP v12.x and v13.x use a different (older) CentOS version, so VLAN tagging works without issue on those releases.
Impact:
-- The system does not prevent you from configuring tagged VLANs, even though they do not pass traffic.
-- Although upgrades complete and you can reboot into the new boot location (or you can set up on Hyper-V from scratch), traffic does not pass (into the guest) across VLANs that are tagged.
Important: If using tagged VLANs on VE setups on Hyper-V is critical to your configuration, you might want to elect to postpone upgrading from a working, v12.x and v13.x release.
Workaround:
Essentially, there is no workaround in this release; you must reconfigure the virtual machine to use separate, untagged interfaces for each VLAN.
Note: Although this is technically a problem between Hyper-V and the built-in LIS on RHEL 7.3/7.5, this issue is being tracked internally in this bug.
Component: TMOS
Symptoms:
After setting a device's ConfigSync IP to 'none' and then back to an actual IP address, the device remains in a disconnected state, and cannot establish ConfigSync connections to other BIG-IP systems in its trust domain.
MCPD periodically logs messages in /var/log/ltm:
err mcpd[27610]: 0107142f:3: Can't connect to CMI peer a.b.c.d, TMM outbound listener not yet created.
Conditions:
--- BIG-IP system is in a trust domain with other BIG-IP systems.
--- Local device's ConfigSync IP is set to 'none', and then back to an actual IP address.
Impact:
Devices unable to ConfigSync.
Workaround:
This workaround will disrupt traffic while TMM restarts:
1. Ensure the local ConfigSync IP is set to an IP address.
2. Restart TMM:
bigstart restart tmm
This workaround should not disrupt traffic:
Copy and paste the following command into the Advanced Shell (bash) on a BIG-IP system, and then run it. This sets the ConfigSync IP for all device objects to 'none', and then back to their correct values.
TMPFILE=$(mktemp -p /var/tmp/ ID882609.XXXXXXX); tmsh -q list cm device configsync-ip > "$TMPFILE"; sed 's/configsync-ip .*$/configsync-ip none/g' "$TMPFILE" > "$TMPFILE.none"; tmsh load sys config merge file "$TMPFILE.none"; echo "reverting back to current"; tmsh load sys config merge file "$TMPFILE"
Component: Local Traffic Manager
Symptoms:
In some unsupported environments, the underlying sock driver uses only only 1 queue. You can confirm whether it does so by executing the tmsh command to check the rxq column (which shows 0):
tmctl -d blade -i tmm/ndal_rx_stats' and
You can verify this on the tx side as well.
Conditions:
This occurs in certain unsupported environments.
Note: When you run 'ethtool -l', you can see: 'command not supported'.
Impact:
When multi-q is present, the use of single queue can impact performance when using the sock driver.
Workaround:
Use other available drivers.
You can check the available drivers by executing the tmsh command:
tmctl -d blade -i tmm/device_probed
Component: Access Policy Manager
Symptoms:
When multiple rate-limiting agents share the same rate-limiting key config, removing one agent may cause other agents to not function properly. In case of using tmm.debug, it may generate a core.
Conditions:
-- Multiple rate-limiting agents sharing the same rate-limiting key config.
-- Removing one agent.
Impact:
Tmm restarts. Traffic disrupted while tmm restarts.
Workaround:
Do not share the same rate-limiting key config among multiple agents.
Component: Application Security Manager
Symptoms:
Live Update modifications are allowed for Application Security Editor Role.
Conditions:
Login as Application Security Editor user and try to install ASU.
Impact:
Application Security Editor Role role is permitted to update Attack Signatures when it shouldn't be.
Component: Local Traffic Manager
Symptoms:
One thread of pkcs11d consumes 100% without any traffic.
Conditions:
-- The BIG-IP system is licensed with NetHSM, and service pkcs11d is running.
-- The MCDP service is restarted.
Impact:
NetHSM configurations and statistics updates are not updated.
Workaround:
Restart the pkcs11d service:
tmsh restart sys service pkcs11d
Component: Advanced Firewall Manager
Symptoms:
AFM FQDN rule matching is broken when multiple FQDN's in firewall policies resolve to the same IP address
Conditions:
-- A firewall policy contains multiple rules
-- Two or more rules point to different fully qualified domain names (FQDNs).
-- Both FQDNs resolve to the same IP address
Impact:
Firewall rule action won't be correctly applied to traffic, causing some traffic to be processed incorrectly by BIG-IP.
Workaround:
Configure FQDN's in such way that no two FQDN's will resolve to the same IP address. If you need to use multiple FQDNs with the same IP address in a policy, configure one as FQDN and the other as a resolved IP address.
Component: Access Policy Manager
Symptoms:
If Network Access resource is being accessed using user interface language other than English, JavaScript errors may be shown in VPN client status window.
Conditions:
- Access Policy with languages other than English
- Network Access resource assigned to this Access Policy
- standalone VPN client in non-English environment
Impact:
VPN connection cannot be established.
Component: TMOS
Symptoms:
There are intermittent auth failures when accessing the BIG-IP administration interfaces via SSH or the GUI.
Conditions:
-- Remote LDAP auth is configured.
-- An idle timeout RST is received on the LDAP connection before the configured auth LDAP idle-timeout expires. This RST might be generated by tmm (if the connection to the LDAP server is via a defined VLAN), some other intervening device on the network, or from the LDAP server itself (depending on its connection time limit).
Impact:
There might be intermittend auth failures.
Workaround:
Set the auth ldap idle-timeout to a smaller value, for example, via tmsh:
modify auth ldap system-auth idle-timeout 299
Component: Local Traffic Manager
Symptoms:
When attaching the port-list to virtual server dest:port-list, the route domain of the virtual server is changed to the default value of 0, and the port-list is not correctly applied. This is encountered in the GUI but not in the CLI.
Conditions:
Using port-list along with virtual server in non default route domain using the GUI.
Impact:
You are unable to use the GUI to attach a port-list that uses a non-default route domain to a virtual server.
Workaround:
Use tmsh to attach a port-list to a virtual server if the port-list uses a non-default route domain.
Component: Local Traffic Manager
Symptoms:
Some received packets are retransmitted back on the incoming VLAN interface.
Conditions:
The symptom is found with the following conditions:
1. A forwarding virtual server is configured.
2. A packet is received whose destination MAC address is its unicast VLAN MAC address and the destination IP address is the broadcast address of that subnet.
Impact:
Broadcast packets are forwarded back to the incoming VLAN interface might result in loops if there are multiple gateways on the network.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
The iRule URI commands are designed to parse a given URI string to each components such as scheme (URI::protocol) or authority (URI::host). The URI::query command is designed to return the query part of an URI, but the returned string contains the fragment part. For example, for the URI "foo://example.com:8042/over/there?name=ferret#nose" (an example from Section 3, RFC 3986), URI::query returns "name=ferret#nose". The "#nose" part should not be present in the return value
Conditions:
Create a test rule with URI having '#' like this.
when HTTP_REQUEST {
# from RFC 3986 Section 3
set url "foo://example.com:8042/over/there?name=ferret#nose"
log local0. "query: [URI::query $url]"
Impact:
URI operations that involve #fragments may fail.
Workaround:
Component: Device Management
Symptoms:
The system generates and logs the following message continuously, at the rate of 3 times a minute, in /var/log/audit:
-- bigip1 notice tmsh[47755]: 01420002:5: AUDIT - pid=47755 user=root folder=/ module=(tmos)# status=[Command OK] cmd_data=cd / ;
-- bigip1 notice tmsh[47755]: 01420002:5: AUDIT - pid=47755 user=root folder=/ module=(tmos)# status=[Command OK] cmd_data=list cm
Conditions:
This occurs during normal operation.
Impact:
Audit log file contains numerous 'cmd_data=list cm device recursive' messages.
Workaround:
Edit the 'include' section of syslog configuration to suppress audit logs of 'cmd_data=cd /' and 'cmd_data=list cm device recursive'.
# tmsh edit /sys syslog all-properties
Replace 'include none' with following syntax.
sys syslog {
- snip -
include "
filter f_audit {
facility(local0) and match(AUDIT) and not match(\"cmd_data=list cm device recursive|cmd_data=cd /\");
- snip -
If you want to filter the message sent to existing remote syslog server,
Set sys syslog remote-servers none,
# tmsh modify sys syslog remote-servers none
Define remote syslog server in "sys syslog include" statement,
Add following filter,
then all message exceludes the message which matches the filter will be sent to remote syslog server.
filter f_remote_server {
not (facility(local0) and message(\"AUDIT\") and match(\"cmd_data=list cm device recursive|cmd_data=cd /\"));
Example configuration is following.
sys syslog {
include "
filter f_remote_server {
not (facility(local0) and message(\"AUDIT\") anD match(\"cmd_data=list cm device recursive|cmd_data=cd /\"));
destination d_remote_loghost {
udp(\"10.0.0.1\" port(514));
log {
source(s_syslog_pipe);
filter(f_remote_server);
destination(d_remote_loghost);
Component: TMOS
Symptoms:
If the virtio driver fails to initialize, it may core during shutdown.
Conditions:
-- Using the virtio VE driver.
-- The virtio driver fails initialization and shuts down instead.
Impact:
TMM cores during driver shutdown.
Component: Global Traffic Manager (DNS)
Symptoms:
Creating WideIP with aliases at the same time causes ZoneRunner to create CNAME RRset without matching A RRset on the peer.
Conditions:
Creating WideIP with aliases at the same time(using GUI or tmsh) causes ZoneRunner to create CNAME RRset without matching A RRset on the peer.
Impact:
GTM peer will not respond with correct answer for DNS request.
Workaround:
Create wideip with two steps.
Component: Access Policy Manager
Symptoms:
'plugin' subsystem of TMM leaks memory, when "HTTP Connector" agent performs DNS query.
Conditions:
Access Policy contains "HTTP Connector" agent.
Impact:
Roughly 500 KB is leaked for every 10000 requests.
Workaround:
None.
Component: TMOS
Symptoms:
Config load fails with an error:
01071769:3: Decryption of the field (privatekey) for object (12004) failed.
Unexpected Error: Loading configuration process failed.
Conditions:
-- BIG-IP configuration has a secured attribute, for example an encrypted dynad key
-- The master key password is changed
-- The configuration is loaded before saving the changes
Impact:
"tmsh load sys config" fails.
Workaround:
After modifying the master key password, save the configuration and then perform the tmsh load sys configuration.
Component: TMOS
Symptoms:
Users running tcpdump with the 'ssl:v' flag to obtain the early traffic secret are given the early master secret instead.
Conditions:
Run tcpdump with the 'ssl:v' flag.
Impact:
Users cannot decrypt TLS1.3 early data packets.
Workaround:
None.
Component: TMOS
Symptoms:
When resolving FQDN names for FQDN nodes/pool members, pending DNS requests are timed out after 5 seconds with no response from the DNS server.
If there is a persistent latency of 5 seconds or greater in the DNS server responses, FQDN name resolution will fail and ephemeral nodes/pool members will not be created.
Conditions:
- BIG-IP using FQDN nodes/pool members
- Persistent latency of 5 seconds or greater in the DNS server responses
Impact:
Ephemeral pool members may not be created, thus no traffic will be sent to the intended pool members.
Workaround:
Resolve any persistent latency issues that might cause delays of 5 seconds or more in DNS server responses.
Component: Access Policy Manager
Symptoms:
During processing of SAML SSO single logout (SLO) requests, a block of tmm memory may become corrupted.
Conditions:
- BIG-IP system is configured as SAML SP.
- External SAML IdP sends SLO request.
Impact:
Various possible negative effects, including TMM core. Traffic disrupted while tmm restarts.
Workaround:
None.
Component: Global Traffic Manager (DNS)
Symptoms:
When importing a BIND zone file, $ORIGIN is appended for rdata from SRV and NAPTR RRs, also not appended for DNAME's owner label.
Conditions:
$ORIGIN is used in original zone files and use zone runner to import.
Impact:
Zone files are not generated correctly.
Workaround:
Do not use $ORIGIN.
Component: TMOS
Symptoms:
Network map shows error message: One or more profiles are inactive due to unprovisioned modules.
Conditions:
-- ASM provisioned.
-- A profile is attached to a virtual but the module supporting the profile is not provisioned
Impact:
The Network Map shows an error message.
Workaround:
Provision the module that supports the profile.
Component: Global Traffic Manager (DNS)
Symptoms:
In v13 later, iRule command RESOLV::lookup @<vs name> does not work
Conditions:
The virtual server targeted by the RESOLV::lookup command is not configured with IP protocol UDP ("ip-protocol udp" in TMSH)
Impact:
The DNS virtual is working while iRule RESOVE::lookup does not send request to the virtual.
Workaround:
Either of these:
1. Modify the target virtual server to specify an IP protocol of UDP
2. Replace the name with IP address to workaround the issue.
set hostip [RESOLV::lookup @198.51.100.1 -a $hostname]
Component: TMOS
Symptoms:
Change not updated in LDAP when the system auth source ('systemauth.source' DB key/'Auth Source Type') is set to Active Directory.
This change is not applied when the setting is modified (e.g., from local or LDAP to Active Directory, or from Active Directory to LDAP). Instead, the change is applied only when MCPD is rewriting the file for other reasons.
Conditions:
Changing the 'systemauth.source' DB key/'Auth Source Type':
-- From local to Active Directory.
-- From LDAP to Active Directory.
-- From Active Directory to LDAP.
Impact:
LDAP data is not updated consistently, and authentication might fail.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
The iRule command LB::down is supposed to send an immediate monitor probe, but it does not.
Conditions:
-- Executing LB::down in an iRule.
Impact:
A monitor probe is not immediately sent, which may cause a pool member to be marked down longer than it should be.
Component: TMOS
Symptoms:
DNS queries are not being cached on the BIG-IP device.
Conditions:
-- DNS cache is enabled (System :: Configuration : Device : DNS Cache).
-- Device receives DNS queries.
Impact:
DNS queries are forwarded, but the BIG-IP system does not cache them.
Workaround:
None.
Component: TMOS
Symptoms:
If a cert is created with a name that begins with * (asterisk) and ending with .crt, you cannot delete it using the GUI.
Conditions:
-- Certificate with a name similar to *example.crt.
-- Select the checkbox in the GUI and click Delete.
Impact:
GUI displays the message: No records to display. The '*example' certificate is still present.
Workaround:
You can use TMSH to delete it without issue.
Component: TMOS
Symptoms:
If you modify an address list associated with a virtual server, any modifications done to virtual addresses are lost when the address list is modified.
Conditions:
This occurs in the following scenario:
-- Create an address list.
-- Assign it to a Virtual Server.
-- Modify some or all of virtual addresses.
-- Modify the address list.
Impact:
Modifications made to virtual addresses are lost.
Workaround:
None.
Component: Global Traffic Manager (DNS)
Symptoms:
When running a debug TMM, if a DNS lookup takes more than 30 seconds, TMM may assert with a message similar to the following in the /var/log/ltm file:
-- notice panic: ../modules/hudfilter/3dns/cache_resolver.c:2343: Assertion "standalone refcnt must be one" failed.
Conditions:
-- Using the debug TMM.
-- Executing a DNS lookup that expires.
Impact:
TMM crash and (in a high availability (HA) configuration) failover.
Workaround:
Do not use the debug TMM.
Component: Local Traffic Manager
Symptoms:
Nitrox5 SSL card failure on a vCMP guest deployed on i11000 platform might cause all SSL transactions to fail.
Conditions:
- i11000 platform.
- vCMP guest.
- Nitrox5 card experiences a failure.
Impact:
- SSL transactions do not complete the handshake.
- Following logs can be seen in /var/log/ltm :
01260013:4 SSL Handshake failed for TCP 10.1.1.5:55368 -> 10.1.1.55:443
01260009:4: 10.2.36.5:55384 -> 10.1.1.1:443: Connection error: ssl_hs_vfy_vfydata_cont:14608: alert(47) verify failed
Component: Policy Enforcement Manager
Symptoms:
PEM subcriber lookup can fail for internet side new connections, as PEM might use the remote address to look up the session, which is not the subscriber.
Conditions:
-- PEM enabled and configured
-- Subscriber session has multiple IP's
-- Each IP lands on a different tmm
Impact:
PEM subscriber lookup can fail on the internet side
Workaround:
No workaround.
Component: Application Security Manager
Symptoms:
It is possible to create certain domain matches with leading dot '.' in tmsh, but not in the GUI.
Conditions:
Advanced WAF bot signature configuration with domain with a leading . character.
Impact:
You are unable to use the GUI to create custom bot-defense signatures.
Workaround:
Use tmsh to add custom bot-defense signatures as follows:
tmsh create security bot-defense signature ockerdocker category Crawler domains add {.ockerdocker} rule "headercontent:\"Google_Analytics_Snippet_Validator\"; useragentonly; nocase;"
Component: Access Policy Manager
Symptoms:
TMM may crash if traffic is run through APM per-request policy containing an empty variable assign agent.
Conditions:
Client traffic is run through virtual server with APM per-request policy that contains empty variable assign agent.
Impact:
TMM may crash. Traffic disrupted while tmm restarts.
Workaround:
None.
Component: Access Policy Manager
Symptoms:
HTTP authentication in the access policy times out after 60 seconds, where previously, the timeout was 90 seconds.
Conditions:
Encountering the timeout of HTTP authentication in the access policy in this version of the software.
Impact:
HTTP authentication times out 30 seconds earlier than it did in previous versions. There is no way to configure this timeout value, so authentication fails for operations that require greater than 60 seconds to complete.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
When a recv string is used with an http monitor, the http status code is collected and in the event of failure, the most recent value (from before the failure) is retrieved and used as part of the log output. This can result in a message that is misleading.
Conditions:
Configure a BIG-IP to monitor an HTTP server.
Impact:
Misleading log messages, difficulty in identifying the real cause of the monitor failure.
Component: Advanced Firewall Manager
Symptoms:
The GUI for device vector is missing functionality to check, add, and delete FQDNs.
Conditions:
This is encountered in the GUI in the DNS tab while viewing the DNS NXDOMAIN Query vector.
Impact:
You are unable to configure a valid FQDN list.
Workaround:
Use TMSH to configure a valid FQDN list.
Component: Traffic Classification Engine
Symptoms:
Beginning in BIG-IP software v14.1.0, Traffic Classification auto signature update fails when performed using the GUI.
The system reports an error:
Error: Exception caught in the script. Check logs (/var/log/hitless_upgrade.log) for details.
Conditions:
Performing Traffic Classification auto signature update using the GUI.
Impact:
Fails to update the classification signature automatically.
Workaround:
You can use either of the following:
-- Perform Traffic Classification auto signature update operations from the CLI.
-- Use the GUI to manually update Traffic Classification signatures.
Component: Local Traffic Manager
Symptoms:
When BIG-IP is configured with at least two VLANs/interfaces, and a virtual server with auto-lasthop disabled, then when that virtual server receives a SYN from a client and sends the SYN/ACK back to the client on a different VLAN/interface, it currently expects the ACK to be received on the outgoing interface.
Conditions:
BIG-IP is configured with (at least) two VLANs/interfaces, and with a virtual server with auto-lasthop disabled.
Impact:
The mismatch could lead to connections failing to establish.
Workaround:
Use the same VLAN on the client side.
Component: Global Traffic Manager (DNS)
Symptoms:
DNS response recursion desired (rd) flag does not match the DNS query when using the iRule command DNS::header rd.
Conditions:
-- iRule command DNS::header rd is used to set DNS query rd bit to a different value.
-- At least one wide IP is configured.
Impact:
DNS response rd flag does not match the DNS query. This is not RFC compliant.
Workaround:
Do not configure any wide IPs.
Component: Local Traffic Manager
Symptoms:
Policy matching may fail to work as expected
Conditions:
Having many conditions with the same operand may trigger an issue where the wrong transition is taken.
This may also be triggered by very complex policies with large numbers of rules.
Impact:
LTM policy matching does not work as expected.
Workaround:
None.
Component: TMOS
Symptoms:
Once Hardware evicts the EPVA TCP flows due to Idle timeout, tmm does not reinsert the flows back when it receives a packet belonging to that flow.
Conditions:
This occurs when the TCP connection is kept idle for ~20 seconds.
Impact:
Performance can be impacted. Hardware acceleration will be disabled for the flows once the flow is removed from hardware due to Idle timeout.
Component: Local Traffic Manager
Symptoms:
TMM stats are reported incorrectly. For example, the system may report double the number of running TMMs or an incorrect amount of available memory.
Conditions:
Changing the DB key merged.method from fast_merge to slow_merge.
Impact:
Incorrect reporting for TMM stats.
Workaround:
Remove the file /var/tmstat/cluster/blade0-performance.
These files are roll-ups and will be re-created as necessary.
Component: Local Traffic Manager
Symptoms:
MCPD crashes when deleting a virtual server and traffic-matching-criteria in the same transaction. This can happen when explicitly using a transaction, or when using a feature that uses transactions, such as when deleting an iApp instance that created these objects.
Conditions:
-- Using a virtual server that has traffic-matching-criteria (e.g., port lists or address lists) attached.
-- Deleting the virtual server and its traffic-matching-criteria in the same transaction.
Impact:
MCP cores, which causes a failover (in a high availability (HA) system) or temporary outage.
Workaround:
Delete the traffic-matching-criteria object separately from the virtual server.
Component: Local Traffic Manager
Symptoms:
Intermittent failure of standby connection mirroring TLS1.3 handshake.
Conditions:
TLS1.3 and connection mirroring. More easily reproduces with ecdsa signature.
Impact:
Standby device fails tls handshake, active success so connection succeeds but not mirrored.
Component: Local Traffic Manager
Symptoms:
Some HTTP/3 streams are terminated with a FIN before all the requested data is delivered.
Conditions:
-- Send multiple HTTP3 requests on different streams simultaneously.
-- The back-end in server is NGINX.
Note: This might happen with other web servers as well.
Impact:
Data transfer is incomplete.
Workaround:
Update the server-side connection to HTTP/2.
Component: Access Policy Manager
Symptoms:
APM has a hosted content feature where files can be uploaded for serving out to end users. This is typically used for Edge Client hosting, CSS, and Images.
When you upload a CSS file with Hosted Content enabled, BIG-IP classifies it as 'text/x-asm' when it should be 'text/css'.
Conditions:
APM Hosted Content feature serving CSS files.
Impact:
Incorrect content type used for CSS may confuse some web clients.
Workaround:
To work around this issue, use the Hosted Content GUI to change the file type from 'text/x-asm' to 'text/css'.
Component: Global Traffic Manager (DNS)
Symptoms:
iRule command DNS::header rd not working as expected.
Conditions:
Virtual server configured with an iRule command to set DNS::header rd.
Impact:
The DNS::header rd iRule command does not set the Recursion Desired flag in DNS headers.
Workaround:
None.
Component: Application Security Manager
Symptoms:
After an action that affects thousands of objects, a subsequent Apply Policy may be missed by a peer.
Conditions:
-- Devices are in an auto-sync device group with ASM sync enabled.
-- A bulk action that affects thousands of objects is performed (e.g., enforcing or disabling all signatures).
-- An Apply Policy action is taken immediately afterwards.
Impact:
Peer devices that are still busy processing the large request miss the Apply Policy action, and it is never sent again.
Workaround:
Make a spurious change and reapply the policy.
Component: TMOS
Symptoms:
The 'bigstart restart bigstart' command shuts down the system without displaying or informing the BIG-IP system user that this command can interrupt service. The system goes directly to the inoperative state as soon as the command is run.
Conditions:
-- Running the command bigstart restart bigstart.
-- Running 'systemctl restart systemd-bigstart' twice.
Impact:
Different versions appear to have different behavior:
-- v12.1.5: shell hangs on bigstart command, but the BIG-IP system stays Active.
-- v13.1.0.7: The BIG-IP system goes inoperative upon 'bigstart restart bigstart'.
-- 1v4.1.2.3: The 'bigstart restart bigstart' command cannot find the 'bigstart' service, but 'systemctl restart systemd-bigstart' shows this behavior.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
With hardware syncookies enabled, HTTP POST requests that are fragmented into separate segments are processed by different TMMs.
Connection is subsequently reset with a TCP RST cause reported as: No flow found for ACK.
Conditions:
-- Hardware syncookies triggered.
-- IP fragmented HTTP POST request.
Impact:
Connection is subsequently reset with TCP RST, cause 'No flow found for ACK'.
Workaround:
None.
Component: Advanced Firewall Manager
Symptoms:
TMM occasionally restarts when running heavy workloads. The crash is a timing-related issue between different tmm threads, and thus happens only occasionally.
Conditions:
-- AFM is provisioned with DoS functionality.
-- The BIG-IP system is under heavy workload.
Impact:
Traffic disrupted while tmm restarts.
Workaround:
None.
Component: Advanced Firewall Manager
Symptoms:
In the BIG-IP GUI, the Network Firewall Active Rule page is blank and nothing is visible.
Conditions:
This occurs when viewing the Active Rule page.
Impact:
You are unable to view active Rules in GUI.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
The ntlmconnpool process continuously restarts after reinstalling the licence. The system reports a message in the BIG-IP console:
Re-starting ntlmconnpool.
Conditions:
This occurs when you upgrade your license such that the new license changes the number of TMMs available.
Impact:
System requires reboot, and reports 'Re-starting ntlmconnpool' message continuously in the BIG-IP console.
Workaround:
You must reboot to clear the condition. Once the system comes back up, it operates as expected.
Component: Local Traffic Manager
Symptoms:
HTTP/2 protocol can be negotiated with the Application-Layer Protocol Negotiation (ALPN) on the Transport Layer Security (TLS) level of communication. When an iRule disables HTTP/2 on a server side, it is assumed that the BIG-IP system no longer offers h2 to a server as an option.
Conditions:
-- A virtual server has an HTTP/2 profile configured on both the client and server sides.
-- A server SSL profile is configured on the virtual server.
-- An iRule using the 'HTTP2::disable serverside' command is attached to the virtual server.
Impact:
The BIG-IP system offers h2 as an option in ALPN when the HTTP/2 profile is disabled on a server side. If h2 is accepted by the server, communication fails since HTTP/2 is disabled and does not decode HTTP/2 traffic.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
The BIG-IP system provides an iRule command 'HTTP2::disable serverside' to put http2 in passthrough mode. When the command is called during the CLIENT_ACCEPTED event, it should completely disable http2 until the end of TCP connection, or until the HTTP2::enable command is executed.
Conditions:
-- A virtual server has an HTTP/2 profile configured on the server side.
-- An iRule with an the command 'HTTP2::disable serverside' command is attached to the virtual server in the CLIENT_ACCEPTED event.
Impact:
The BIG-IP system continues to send HTTP/2 traffic to a server.
Workaround:
None.
Component: TMOS
Symptoms:
When the Management IP address assignment is changed and the IP address obtained from DHCP lease is used for static interface configuration, the management port might become unreachable after the DHCP lease expiration time, even though interface has a static IP configured.
Conditions:
-- Management IP assignment is changed from dynamic (DHCP) to static.
-- The static IP address that is configured is identical to the DHCP address that was assigned.
Impact:
Remote management access is lost after the DHCP lease expires.
Workaround:
When changing the management interface configuration from DHCP to static, first delete the old configuration, then create new configuration. This can be done with TMSH:
(tmos)# modify sys global-settings mgmt-dhcp disabled
(tmos)# del sys management-ip 10.14.30.111/24
(tmos)# create sys management-ip 10.14.30.111/24 { description configured-statically }
Component: Application Visibility and Reporting
Symptoms:
Discrepancy in AVR reports. When filtering on the 'last month' interval, a specific number of total requests per virtual server is shown. Then when filtering to the present day from a date that encompasses that month, a lower number is reported.
Conditions:
-- Number of records in database exceeds the maximum mount of data that AVR can aggregate between different table-resolutions.
-- There are metrics on the report other than the default one (hits-count).
Impact:
Stats on DB get corrupted and incorrect.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
When BIG-IP is configured with transparent vlan-group and traffic is matching a standard or fastl4 virtual-server and traffic hitting BIG-IP does not have a destination MAC address that belongs to BIG-IP - traffic will be L2 forwarded and pool member selection will not happen.
This defect will also cause active FTP data connections over vlan-group to fail.
Conditions:
All conditions must be met:
- Traffic over transparent vlan-group.
- Standard or fastl4 virtual-server.
- Traffic has a destination MAC address that does not belong to BIG-IP.
- Standard virtual server with FTP profile is configured.
- Active FTP session is in use.
- Traffic flows over vlan-group.
Impact:
Server-side connections will fail.
Workaround:
Use opaque vlan-group instead.
disable db variable connection.vgl2transparent (15.0+)
Component: Application Security Manager
Symptoms:
When downloading and installing the latest ASU file manually the Live Update indicator located at the top left of the screen still indicates that there is a new update available.
Conditions:
-- The Live Update scheduler is not in auto mode (System :: Software Management :: Live Update :: Installation of Automatically Downloaded Updates = Disabled).
-- Upload and update the latest ASU file manually.
Impact:
The Live Update indicator continues to indicate on a new update though the latest file was installed.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
The SSL profile "passive-close" option is available in TMSH (but not the GUI), but is not actually used.
A side-effect of this issue is: when the "passive-close" option is configured in TMSH, if the profile is later modified in the GUI, the "passive-close" option will be removed from the profile.
Conditions:
-- Modifying a client or server SSL profile in TMSH.
Impact:
The "passive-close" option is not actually used.
Workaround:
Do not use the "passive-close" option.
Component: Local Traffic Manager
Symptoms:
BIG-IP systems provide manipulation tools over a connection with an LTM policy and/or iRule. LTM policy takes precedence over iRules and has an option to shutdown a connection based on satisfied conditions. When a connection is closing, an iRule should not be executed under the same conditions.
Conditions:
-- The BIG-IP system has a virtual server with an LTM policy and an iRule.
-- The LTM policy has action 'shutdown connection' under certain conditions.
-- The iRule has an event which is triggered under the same conditions.
Impact:
The iRule is executed before the connection is being reset.
Workaround:
None.
Component: Application Security Manager
Symptoms:
When overwriting a parent policy with import/replace, elements from the parent policy that were deleted remain in the child policies.
Conditions:
-- A parent policy exists with a child policy that inherits a section in which new configuration elements can be created in the parent policy (like ip address exceptions).
-- An element is deleted from the parent policy, and then the parent policy is exported.
-- The parent policy is then imported to replace a parent policy on a different device to perform the same changes on its children.
Impact:
The children on the different devices are left unexpectedly in different states.
Component: TMOS
Symptoms:
When BGP peer is going down, the BIG-IP system sends the wrong 'bgpPeerState: 6(established)' with its SNMP trap.
Conditions:
-- BIG IP system is connected with a Cisco router to verify the traps.
-- BGP peer between the BIG-IP system and the Cisco router is going down.
-- Both devices release an SNMP trap.
Impact:
The BIG-IP system sends the wrong code with its SNMP trap. It should be 'bgpPeerState: idle(1)' when the peer is not connected.
Workaround:
None.
Component: Application Security Manager
Symptoms:
Remote syslog server is unable to properly parse the violation detail buffers as UTF-8.
Conditions:
This occurs when the violation detail buffers contain double-byte/non-UTF characters, due to requests that contain non-ASCII UTF-8 characters.
Impact:
The syslog server cannot parse violation detail buffers as UTF-8.
Workaround:
None.
Component: Access Policy Manager
Symptoms:
Client JavaScript may see a non-normalized URL for the IFRAME HTML element in Portal Access.
Conditions:
- Original HTML page contains IFRAME element with relative URL
- JavaScript code reads this URL from the IFRAME element
Impact:
The URL for IFRAME element is not normalized and the web application may not work correctly in Portal Access.
Workaround:
Use an iRule to correct the returned IFRAME URL in rewritten JavaScript code. There is no generic iRule pattern; the correction depends on actual JavaScript code, but you can use the following example as a template:
when REWRITE_REQUEST_DONE {
if { [HTTP::path] ... } { # use appropriate selection for URI
set correct_location 1
when REWRITE_RESPONSE_DONE {
if {[info exists correct_location]} {
unset correct_location
# look for the piece of code to be corrected
set str2find {...} # use appropriate pattern for rewritten code
set str_len [string length $str2find]
set strt [string first $str2find [REWRITE::payload]]
# make replacement using appropriate corrected code
if {$strt > 0} {
REWRITE::payload replace $strt $str_len {...}
Component: Application Security Manager
Symptoms:
If the ASM http-methods are missing from the MCP configuration, importing an XML ASM policy creates a policy that has no allowed methods and will block all traffic.
Conditions:
-- BIG-IP system configuration is loaded without the required asm_base.conf.
-- An XML ASM policy is loaded.
Impact:
All traffic is blocked for the policy.
Workaround:
Recreate the required methods (GET, POST, etc.) in the policy.
Component: Advanced Firewall Manager
Symptoms:
When loading a configuration, the config load fails with an error:
Invalid self IP, the IP address <ip_addr> already exists.
Conditions:
-- Config contains an IPv4 SelfIP
-- Config contains an IPv4-mapped IPv6 address that is assigned to the same vlan
BIG-IP does not prevent you from creating this condition and will allow you to save it.
Impact:
During configuration load will fail:
0107176c:3: Invalid self IP, the IP address <ip_addr> already exists.
Unexpected Error: Loading configuration process failed.
Workaround:
Delete one of the SelfIP addresses and load the configuration.
Component: TMOS
Symptoms:
When setting "SystemMaxUse" to any value, systemd does not get honored and the specified size is exceeded
Conditions:
-- Using a Non-TMOS user account with external authentication permission.
-- Systemd-journald is configured to create a user journal for every user that logs into the BIG-IP system.
Impact:
Journald filling up file system size. These journals are allocated with a minimum size of 4MiB and are not removed when the log entries age-out.
Workaround:
Remove journal logs manually.
Component: TMOS
Symptoms:
When attempting to use TFTP or Active FTP at the BIG-IP management port to transfer files to a remote system, the connection eventually times out and the file is not transferred.
This is expected behavior resulting from the enhancement made in BIG-IP v14.1.0:
"Support for network firewall rules on the management port" :: https://techdocs.f5.com/kb/en-us/products/BIG-IP_ltm/releasenotes/product/relnote-bigip-14-1-0.html#rn_ltm-tmos_1410_new.
When attempting to use TFTP and Active FTP via tmm interfaces will work as it has the necessary Algorithm capabilities to set up return listeners.
Conditions:
- BIG-IP v14.1.0 or greater.
- Attempt to initiate TFTP or Active FTP from the BIG-IP management port through command line.
Impact:
Unable to use TFTP or Active FTP to transfer files to/from the BIG-IP system over management port
Workaround:
Consider using encrypted transport (sftp, scp, etc.) in order to avoid the exposure of sensitive data, including passwords.
Manually load connection tracking for the necessary protocol(s) from the command line with:
modprobe nf_conntrack_ftp
modprobe nf_conntrack_tftp
Component: Access Policy Manager
Symptoms:
Event handler defined on DOM elements is not executed properly, and some events on a page like onClick() are not executed properly.
Conditions:
-- Portal access enabled.
-- Rewrite enabled.
-- New value of event handler is equal to inline handler already set for the element.
Impact:
Web application does not operate as expected.
Workaround:
Use a custom iRule, for example, an iRule that redefines F5_Inflate_onclick:
Note: In the following iRule, substitute <PATH_TO_PAGE> with the page in your web application.
when REWRITE_REQUEST_DONE {
[HTTP::path] ends_with "<PATH_TO_PAGE>"
# log "URI=([HTTP::path])"
# Found the file to modify
REWRITE::post_process 1
set do_it 1
when REWRITE_RESPONSE_DONE {
if {[info exists do_it]} {
unset do_it
set strt [string first {<script>try} [REWRITE::payload]]
if {$strt > 0} {
REWRITE::payload replace $strt 0 {
<script>
(function(){
var ioc = F5_Inflate_onclick;
F5_Inflate_onclick = function(o, incr, v) {
if (v === o.onclick && typeof v === 'function') {
return v;
return ioc.call(this,o,incr,v);
})();
</script>
Component: Access Policy Manager
Symptoms:
When configuring the OAuth provider and trying to set the task frequency to fewer than 60 minutes, the BIG-IP reports an error:
01b70003:3: Discovery interval (10) for OAuth provider must be greater than (60) minutes.
Conditions:
This occurs when configuring the frequency interval of an OAuth provider to a value lower than 60 minutes.
Impact:
You are unable to create a provider with a frequency interval of fewer than 60 minutes.
Workaround:
Use a value of 60 minutes or higher.
Component: Application Security Manager
Symptoms:
When there is a license change at the same time as a security update (ex. Threat Campaigns or Attack Signatures), the system can reach a deadlock which blocks some operations, eventually leading to all the REST threads becoming blocked and unresponsive.
Conditions:
A license change occurs at the same time as a security update (ex. Threat Campaigns or Attack Signatures).
Impact:
ASM user interfaces are unresponsive.
Workaround:
Kill asm_config_server.pl or restart ASM
Component: TMOS
Symptoms:
Under general usage; WCCP crashes with a "ServiceGroup size exceeded" exception.
Conditions:
Have WCCP service groups configured.
Impact:
WCCP throws an exception and crashes.
Workaround:
None.
Component: Application Security Manager
Symptoms:
TMM may crash when Bot Defense is enabled and network DNS is configured.
Conditions:
This can occur when is Bot Defense enabled and network DNS is configured.
Impact:
Traffic disrupted while tmm restarts.
Workaround:
None.
Component: TMOS
Symptoms:
Certificate-ldap only returns the first matching oid from the certificate file even though multiple matching san-other entries exists
Conditions:
When Certificate-ladp attribute ssl-cname-field set to san-other and certificate with multiple san-other oids
Impact:
Only the first matching oid is returned.
Component: Application Security Manager
Symptoms:
When remote logger is enabled and violation_details field is selected for output, but violations field is not selected - content of violation_details field appears as "N/A".
Conditions:
- Remote logger is enabled;
- violation_details field is selected for output;
- violations field is not selected for output;
- violation is detected and reported to remote logger.
Impact:
Remote logger will not contain violation_details in report.
Workaround:
Enable violations field for remote logging.
Component: Local Traffic Manager
Symptoms:
TMM crashes.
Conditions:
iRule with "SSL::extensions insert"
Impact:
Traffic disrupted while tmm restarts.
Workaround:
Component: Global Traffic Manager (DNS)
Symptoms:
Changing the contents of a topology region record may result in DNS queries temporarily being directed as if the change had not happened for queries from the IP address of the last end user client to use topology load balancing.
Conditions:
-- A client at a single IP address makes multiple queries that are load balanced using topology, both before and after a change to a topology region record, where that change also modifies the result the single client receives.
-- If a query from a different client IP address is received and load balanced using topology, then the issue is corrected until the next change to a topology region record.
Impact:
After changing the contents of a topology region record, the last end user client to send a query before the change may receive the wrong load balancing decision if the change affected that decision. Queries from other end user clients are load balanced correctly and cause the issue to go away until the next topology region record change.
Workaround:
This issue can be temporarily corrected by sending a DNS query that is load balanced using topology after making changes to region records.
Component: Application Security Manager
Symptoms:
-- CPU utilization is high on the odd-numbered cores.
-- Messages appear at 60-second intervals in /var/log/ts/asm_start.log:
update_GTM_score
Conditions:
-- One or more virtual servers have FTP/SMTP/WEBSEC profiles attached to it.
-- ASM configured.
Impact:
Elevated CPU usage.
Workaround:
-- On the BIG-IP system, edit the file /etc/ts/tools/nwd.cfg to change the value EnforcerCpuReportTimeInterval from 60 to a higher value, e.g., 3600 for once an hour, or even larger.
-- Restart ASM:
bigstart restart asm
Component: Local Traffic Manager
Symptoms:
The client-side connection of the dhcpv4_fwd profile on Broadcast DHCP-Relay Virtual Server never expires from the connection table.
Conditions:
Configure dhcpv4_fwd profile on Broadcast DHCP-Relay Virtual Server.
Impact:
Even after correcting the listener to use dhcpv4 (relay) instead of dhcpv4_fwd (forwarding) profile, the client-side connection from the dhcpv4_fwd profile remains.
Workaround:
Delete the long-standing connection from the connection table.
Component: TMOS
Symptoms:
For BIG-IP v14.1.x and later, the default testing page of the Apache web-server is accessible at <mgmt-ip>/noindex.
Conditions:
This is encountered when navigating to the /noindex page from the web browser.
Impact:
Limited information about the Apache web server and its operating system is available to users with access to the mgmt port interface.
Workaround:
None.
Component: Global Traffic Manager (DNS)
Symptoms:
Messages similar to the following may be seen in the DNS (GTM) logs:
The list processing time (32 seconds) exceeded the interval value. There may be too many monitor instances configured with a 30 second interval.
This message was introduced in 15.0.0 as an aid to help identifying overloaded DNS (GTM) systems, but it triggers too easily and can be logged when the device is not overloaded.
Conditions:
-- DNS (GTM) servers are present.
-- Virtual servers are configured on those DNS (GTM) servers.
-- A monitor is applied to the DNS (GTM) server.
Impact:
Messages are logged that imply the system is overloaded when it is not.
Workaround:
Create a log filter to suppress the messages
sys log-config filter gtm-warn {
level warn
message-id 011ae116
source gtmd
Component: Local Traffic Manager
Symptoms:
When QUIC is discovering the link bandwidth, it sometimes increases the congestion window when there is not enough available data to use the existing congestion window.
Conditions:
When the application is not generating enough data, or any streams with data are stream flow control limited.
Impact:
In some cases, the congestion window can be too large for the path and cause packet losses if it is later fully utilized.
Workaround:
Component: Local Traffic Manager
Symptoms:
For the i4x00 and 4000 platforms, egress trunk distribution will be unbalanced if the number of trunk members is not a power of 2.
Conditions:
A trunk is configured with an odd number of trunk interfaces or a trunk member goes down such that the number of working members is odd.
Impact:
Uneven traffic distribution. Some interfaces will see more traffic than others.
Workaround:
Insure the number of trunk interfaces is a power of 2: 2, 4, or 8.
Component: Global Traffic Manager (DNS)
Symptoms:
DNS :: Zones :: ZoneRunner :: Resource Record List :: Search All Records :: click on record of type CAA
The page shows an error:
Resolver returned no such record.
Conditions:
Attempting to manage a CAA record via the GUI
Impact:
Unable to update CAA records via the GUI.
Workaround:
You can use either of the following workarounds:
-- Manually edit the BIND configuration.
-- Delete the record and create a new one with the desired changes.
Component: TMOS
Symptoms:
After running cpcfg and booting into the volume, daemons such as named and gtmd are stuck restarting. Additionally the SELinux audit log contains denial messages about gtmd and named being unable to read unlabeled_t files.
Conditions:
Running cpcfg on a volume that has already been booted into.
Impact:
Services do not come up.
Workaround:
In the bash shell, force SELinux to relabel at boot time. Then reboot:
# touch /.autorelabel
# reboot
Component: Local Traffic Manager
Symptoms:
A configuration with a virtual server-to-virtual server flow established, for example by the 'virtual' iRule command, and using a TCP stack with 'Tail Loss Probe' enabled, might encounter a race between the delayed ACK and the tail loss probe, which can lead to a tmm_panic or an OOPs message:
no trailing data.
Conditions:
-- Virtual server-to-virtual server flow established.
-- TCP profile with 'Tail Loss Probe' enabled.
-- Certain timing related traffic scenario.
Impact:
TMM generates a core and reports an OOPs message:
no trailing data.
Workaround:
Do not use a TCP stack with 'Tail Loss Probe' enabled in conjunction with a virtual server-to-virtual server flow configuration.
Component: Application Security Manager
Symptoms:
When ASM detects "virus" (with help of external icap server), the response page will be JS-Challenge instead of blocking.
Conditions:
-- ASM provisioned.
-- ASM policy attached to a virtual server.
-- Anti-Virus protection enabled in ASM policy.
-- ASM finds a virus in a request.
Impact:
-- End user client gets JS-Challenge response instead of blocking page.
-- End user does not see ASM support ID.
-- Browser can run the JavaScript and resend the request to ASM, which is then forwarded to the backend server.
Workaround:
None.
Component: TMOS
Symptoms:
The missing PAM_RHOST setting causes the radius packet to go out without the calling-station-id avp
Conditions:
1. Configure radius server and add it to BIG-IP
tmsh create auth radius system-auth servers add { myrad }
2. modify auth source type to radius
tmsh modify auth source { type radius }
3. try to authenticate to BIG-IP using iControl REST
Impact:
Remote authentication using iControl REST is not allowed based on calling-station-id
Component: TMOS
Symptoms:
In BIG-IP v10.x it was possible to change the browser cache timeout from bigpipe using the command:
bigpipe httpd browsercachetimeout
In 14.1.2.1 and newer, it is still possible to change the value in the GUI using "System :: Preferences :: Time To Cache Static Files.
However there is no tmsh equivalent in any version.
Conditions:
This is encountered when you try to configure the GUI browser cache timeout setting using tmsh.
Impact:
Unable to modify browser cache timeout except from GUI
Workaround:
Using GUI to configure this field. GUI System :: Preferences :: Time To Cache Static Files.
Component: Application Security Manager
Symptoms:
Horizontal scroll added to the page unnecessarily.
Conditions:
This occurs when viewing the Threat Campaigns or Brute Force Attacks page in any browser
Impact:
The horizontal scroll bar breaks the intended page layout.
Workaround:
Component: Service Provider
Symptoms:
Message Routing Diameter profile does not forward diameter messages that include an AVP with a zero (0) length data field.
Conditions:
-- A virtual server with an Message Routing Diameter Profile.
-- A diameter message containing an AVP with a zero length data field.
Impact:
Diameter messages with zero length AVPs are not forwarded as expected.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
On all versions 12.1.0 or later, if you change the HTTPS port (e.g., to 8443, as is required for '1NIC' BIG-IP Virtual Edition (VE) deployments) and then expose the management UI via a self IP in a non-zero route domain, you cannot access the system via the GUI or CLI, and the system does not pass traffic as expected.
In versions 14.1.0 and later on VE installations, attempting to manage a BIG-IP system over a self IP can fail if all these conditions are met:
-- Non-standard HTTPS port used.
-- No TMM default route configured.
-- No route to the client IP address configured.
Conditions:
-- Modify the default HTTPS and/or default SSH ports.
And either of the following:
On 12.1.0 and above:
-- Expose the management UI and/or CLI via a self IP in a non-zero route domain.
On 14.1.0 and above:
-- No TMM default route configured.
-- No route to the client IP address configured.
Impact:
-- Unable to access BIG-IP GUI on non-standard HTTPS port.
-- Unable to access BIG-IP CLI on non-standard SSH port.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
There can be an undisciplined clock stanza in /etc/ntp.conf, resulting in an undisciplined clock.
NTP documentation:
http://support.ntp.org/bin/view/Support/UndisciplinedLocalClock
Conditions:
This might occur in at least the following ways:
-- No server is specified in 'sys ntp servers {}'.
-- A server does exist, but an improper method was used to configure the NTP server.
Impact:
When the LOCAL undisciplined clock is left as a valid time-source, it delays the system synchronizing time to a real NTP server. It can also result in time being adjusted incorrectly if the the remote time-source becomes unreachable.
Workaround:
Configure a dummy server via 'ntp servers {}' that does not respond.
While this removes the undisciplined local clock, it does result in ntpd having an unreachable time source, and could be flagged in diagnostics, misdirect other troubleshooting, generate unnecessary traffic, etc.
However, if the 'dummy' source starts responding, it could become a rogue time source.
Component: Access Policy Manager
Symptoms:
Tmm crashes after configuring a radius server pool.
Conditions:
-- Radius server pool exists
-- Radius server pool does not have a designated load balancing algorithm.
Impact:
TMM will core while radius accounting stops. Traffic disrupted while tmm restarts.
Workaround:
Component: TMOS
Symptoms:
Loading (with merge) a configuration file that references some iRules results in validating every iRule and ends up validating the same procedures multiple times for every virtual server a single iRule is associated with.
Conditions:
Configuration which has 100's of virtual servers, some iRules that are assigned to all virtual servers and a few library iRules.
Impact:
Task fails (via REST) or ends up taking a really long time when run manually.
Workaround:
None.
Component: TMOS
Symptoms:
After upgrading BIG-IP to 14.1 the LDAP/AD remote authentication will fail .
The /var/log/secure will show :
/secure:
Dec 6 15:27:44 hostname err httpd[9402]: pam_ldap(httpd:auth): error opening connection to nslcd: No such file or directory
Dec 6 15:27:44 hostname notice httpd[9402]: pam_ldap(httpd:auth): auth server unavailable, trying fallback
Dec 6 15:27:44 hostname warning httpd[9402]: pam_unix(httpd:auth): check pass; user unknown
Dec 6 15:27:44 hostname notice httpd[9402]: pam_unix(httpd:auth): authentication failure; logname= uid=48 euid=48 tty= ruser= rhost=192.168.227.145
/var/log/daemon.log will show ;
/daemon:
Dec 6 15:29:40 hostname notice systemd[1]: nslcd.service: main process exited, code=exited, status=1/FAILURE
Dec 6 15:29:40 hostname notice systemd[1]: Unit nslcd.service entered failed state.
Dec 6 15:29:40 hostname warning systemd[1]: nslcd.service failed.
Dec 6 15:35:47 hostname notice systemd[1]: nslcd.service: main process exited, code=exited, status=1/FAILURE
Dec 6 15:35:47 hostname notice systemd[1]: Unit nslcd.service entered failed state.
Dec 6 15:35:47 hostname warning systemd[1]: nslcd.service failed.
> Dec 06 15:35:47 hostname systemd[1]: Started Naming services LDAP client daemon..
> Dec 06 15:35:47 hostname systemd[1]: Starting Naming services LDAP client daemon....
> Dec 06 15:35:47 hostname nslcd[8050]: nslcd: /etc/nslcd.conf:15: usertemplate: too may arguments
> ===================== > This is the hint that user-template is at fault
Conditions:
LDAP/nslcd config , remote authentication , user-template used
The values within user-template include white spaces :
example: uid=%s,CN=my home,OU=Generic Users,OU=good Users,OU=users,DC=users,DC=org
Impact:
LDAP/nslcd process failed with "error opening connection to nslcd" when user-template includes white spaces.
Workaround:
Replace the white-space character with underscore "_" in the user-template if possible, or remove the user-template and restart nslcd daemon
Component: Local Traffic Manager
Symptoms:
Version: 13.1.3.1
# tmsh list ltm profile tcp tcp proxy-buffer-high proxy-buffer-low
ltm profile tcp tcp {
proxy-buffer-high 49152
proxy-buffer-low 32768
proxy-buffer-high
Specifies the highest level at which the receive window is closed.
The default value is 49152.
proxy-buffer-low
Specifies the lowest level at which the receive window is closed.
The default value is 32768.
Version: 14.1.2.2
# list ltm profile tcp TCP proxy-buffer-high proxy-buffer-low
ltm profile tcp tcp {
proxy-buffer-high 65535
proxy-buffer-low 32768
proxy-buffer-high
Specifies the highest level at which the receive window is closed.
The default value is 131072.
proxy-buffer-low
Specifies the lowest level at which the receive window is closed.
The default value is 98304.
Conditions:
Looking at the help for proxy-buffer-high and proxy-buffer-low in tmsh
Impact:
The default value for proxy-buffer-high is 65535 and the default value for proxy-buffer-low is 32768, but the help text indicates that the defaults are 13072 and 98304 respectively.
Component: TMOS
Symptoms:
The SSL Orchestrator configuration is not synced properly across the high availability (HA) configuration.
The REST framework versions are different on the devices.
Conditions:
-- BIG-IP devices configured for HA.
-- SSL Orchestrator configured.
-- Upgrading from v14.1.2 to v15.1.x or newer.
Impact:
SSL Orchestrator configuration does not sync across BIG-IP HA peers.
Workaround:
The following steps are required on all HA, first on the active and then on the standby BIG-IP devices.
1. Open a BIG-IP terminal session with admin/root level access.
2. Run the following commands, in the order specified:
bigstart stop restjavad
rm -rf /shared/em/ssl.crt/*
bigstart start restjavad
restcurl -X DELETE shared/resolver/device-groups/tm-shared-all-big-ips/devices
restcurl -X DELETE shared/gossip-conflicts
restcurl -X DELETE shared/device-certificates
restcurl -X POST -d '{"generateKeyPair": true}' shared/device-key-pair
bigstart restart restjavad restnoded
Component: TMOS
Symptoms:
If you have BIG-IPs in a Device Service Cluster, and you attempt to sync a new floating self-IP over to a standby on a VLAN that the standby does not currently have a non-floating self-IP on, you will get an error and the sync will fail. This is the correct behavior. The issue, though, is that if you subsequently create a non-floating self-IP on the standby in order to rectify this issue, the sync will still fail.
Conditions:
-- BIG-IPs configured in a Device Service Cluster.
-- Device group is configured to use Automatic Sync or Manual with Incremental sync.
-- Attempting to sync a floating self-IP to a system that does not have a non-floating self-IP on the same VLAN.
Impact:
You are unable to sync BIG-IPs. Both devices will be out of sync and you will see an error displayed:
01070355:3: Self IP <address> is declared as a floating address but there is no non-floating address defined for this network
Even after you add a non-floating self-IP on the affected device, a subsequent config sync does not fix the error.
Workaround:
If you make any other configuration change that generates a config sync, this will correct itself after the other device has added a non-floating Self-IP.
Otherwise, this can be corrected by doing a full config sync, and can be done via the GUI or via tmsh.
In the GUI, change the Sync Type for the device group to Manual with Full Sync, and then do a config sync.
In tmsh, the command is:
run cm config-sync force-full-load-push to-group <affected_device_group>
Component: Global Traffic Manager (DNS)
Symptoms:
After adding a new wildcard wideip, DNS requests start matching the wildcard even if a more specific wildcard wideip should match.
Conditions:
New less specific Wildcard WideIPs are created.
Impact:
DNS request matches less specific WideIP.
Workaround:
# tmsh load sys config gtm-only
restart tmm
Component: TMOS
Symptoms:
SSL Orchestrator configuration deployment across BIG-IP devices in a high-availability (HA) group may result in inconsistent state, if during deployment the connectivity between the HA peers is lost.
Conditions:
Deploying SSL Orchestrator configuration across BIG-IP devices in an HA group.
Impact:
Inconsistent SSL Orchestrator configuration on BIG-IP devices in an HA group.
Workaround:
Run the /usr/bin/ha-sync script. See ha-sync -h for help.
Component: Local Traffic Manager
Symptoms:
When you upgrade an 11.x/12.x device with route advertisement enabled, you might discover a difference between the running configuration and the saved configuration post upgrade, which might result in route advertisement becoming disabled.
-- In the running configuration, the virtual-addresses route advertisement setting 'enabled' changes to 'selective'.
-- In bigip.conf, the virtual-addresses route advertisement setting is still set to 'enabled'.
-- After config load or after re-licensing, the virtual-addresses route advertisement reverts to disabled.
Conditions:
-- Upgrading an 11.x/12.x device with route advertisement enabled.
-- After saving the config, both the running-config and bigip.conf have the same value: i.e., 'selective'.
-- Loading the configuration (tmsh load sys config) results in route advertisement becoming disabled.
Impact:
The route-advertisement setting is 'enabled' in the config file, but 'selective' in the running configuration. This has the following impact:
If you save the configuration and then reload it, the route advertisement is changed to 'selective' in the config file and 'disabled' in the running config.
Workaround:
You can identify whether systems running v13.0.0 or higher are at risk of encountering this issue by checking a legacy internal setting, ROUTE_ADVERTISEMENT:
Procedure to identify whether virtual-addresses are affected, that have an incorrect setting in the legacy ROUTE_ADVERTISEMENT artifact:
Virtual-addresses may be affected by this issue on v13.0.0 and higher if ROUTE_ADVERTISEMENT=true in mcpd.
You can check this value with the guishell command:
guishell -c "select NAME,ROUTE_ADVERTISEMENT,RA_OPTION from virtual_address";
Example:
guishell -c "select NAME,ROUTE_ADVERTISEMENT,RA_OPTION from virtual_address";
-----------------------------------------------------------
| NAME | ROUTE_ADVERTISEMENT | RA_OPTION |
-----------------------------------------------------------
| /Common/10.32.101.41 | false | 0 | <<< no risk, virtual-address created in 13.1.3.2 with route-advertisement disabled
| /Common/10.32.101.42 | false | 2 | <<< no risk, virtual-address created in 13.1.3.2 with route-advertisement selective
| /Common/10.32.101.43 | false | 1 | <<< no risk, virtual-address created in 13.1.3.2 with route-advertisement enabled
| /Common/10.32.101.47 | true | 0 | <<< MEDIUM RISK virtual-address from a 11.6.2 upgrade or 11.6.2 ucs with route-advertisement not in use
| /Common/10.32.101.49 | true | 1 | <<< HIGH RISK virtual-address from a 11.6.2 upgrade or 11.6.2 ucs with route-advertisement enabled
Any virtual address that shows ROUTE_ADVERTISEMENT=true is at risk. If true but route-advertisement is not in use, there is no risk until route-advertisement is configured later.
------------------------------------------------------------------------------------------
Procedure to remove the legacy ROUTE_ADVERTISEMENT artifact from the config on systems found to be affected:
1. Review Standby system (if available) and ensure Route Advertisement in running configuration is configured and functioning as desired with "tmsh list ltm virtual-address route-advertisement". If not, manually correct Route Advertisement to desired configuration and confirm functionality.
2. Fail over Active system to Standby status:
tmsh run sys failover standby
3. Review former Active (now Standby) system and ensure Route Advertisement in running configuration is configured and functioning as desired. If not, manually correct Route Advertisement to desired configuration.
4. Save the config to disk:
tmsh save sys config
5. Load the config from disk. This may temporarily cause route-advertisement to revert to disabled on at risk virtual-addresses:
tmsh load sys config
6. Load the config a 2nd time. This removes the legacy artifact, re-enables route-advertisement as per the configuration, and leaves the system in a not-at-risk state:
tmsh load sys config
7. Verify it worked:
guishell -c "select NAME,ROUTE_ADVERTISEMENT,RA_OPTION from virtual_address";
Example of a fixed config:
guishell -c "select NAME,ROUTE_ADVERTISEMENT,RA_OPTION from virtual_address";
-----------------------------------------------------------
| NAME | ROUTE_ADVERTISEMENT | RA_OPTION |
-----------------------------------------------------------
| /Common/10.32.101.41 | false | 0 | <<< no risk, virtual-address created in 13.1.3.2 with route-advertisement disabled
| /Common/10.32.101.42 | false | 2 | <<< no risk, virtual-address created in 13.1.3.2 with route-advertisement selective
| /Common/10.32.101.43 | false | 1 | <<< no risk, virtual-address created in 13.1.3.2 with route-advertisement enabled
| /Common/10.32.101.47 | false | 0 | <<< no risk, virtual-address from a 11.6.2 upgrade or 11.6.2 ucs with route-advertisement not in use
| /Common/10.32.101.49 | false | 1 | <<< no risk, virtual-address from a 11.6.2 upgrade or 11.6.2 ucs with route-advertisement enabled
------------------------------------------------------------------------------------------
If you encounter this issue and route-advertisement becomes disabled before cleaning the legacy ROUTE_ADVERTISEMENT artifact from the config, reload the configuration again using the following command to set the running config and saved config to 'selective':
tmsh load sys config
Component: Application Security Manager
Symptoms:
There is no confirmation dialog for applying a policy.
Conditions:
This occurs when visiting any policy page available from the policies list.
Impact:
Without a confirmation dialog, it is easier to apply a policy by mistake
Workaround:
Component: Local Traffic Manager
Symptoms:
TMM keeps restarting after setting a self IP to an IPv6 address with an embedded IPv4 address in TMSH.
Conditions:
Set self IP to an IPv6 address with an embedded Ipv4 address using tmsh.
Impact:
Tmm restarts repeatedly. Traffic disrupted while tmm restarts.
Workaround:
Set self IP to IPv4 address.
Component: TMOS
Symptoms:
With BIG-IP devices configured in high availability (HA) mode, with SSL Orchestrator configured, when upgrading from v14.1.2 to v15.1.x or newer, the SSL Orchestrator configuration is not synced properly across the high availability (HA) configuration.
This problem is caused by a REST framework sync issue between the devices in the high availability (HA) pair.
Conditions:
-- BIG-IP devices configured in high availability (HA) mode.
-- SSL Orchestrator configured.
-- Upgrading from v14.1.2 to v15.1.x or newer.
Impact:
SSL Orchestrator configuration not syncing across the BIG-IP high availability (HA) pair.
Workaround:
The following steps are required on both high availability (HA) peers, first on the active and then on the standby BIG-IP device.
1. Open a terminal session with admin/root level access.
2. Run the following commands, in the order specified:
bigstart stop restjavad
rm -rf /shared/em/ssl.crt/*
bigstart start restjavad
restcurl -X DELETE shared/resolver/device-groups/tm-shared-all-big-ips/devices
restcurl -X DELETE shared/gossip-conflicts
restcurl -X DELETE shared/device-certificates
restcurl -X POST -d '{"generateKeyPair": true}' shared/device-key-pair
bigstart restart restjavad restnoded
Component: Advanced Firewall Manager
Symptoms:
Search from shared objects of IP addresses in the address list and port numbers in the port list does not work as expected. In previous releases, it was possible to expand all address/port lists in a single operation. There is no support for that in this release.
Conditions:
Searching from shared objects of IP addresses in the address list and port numbers in the port list.
Impact:
It is no longer possible to expand all address/port lists in a single operation, and because all lists needs to be expanded to allow a search using the browser or cache the objects in the embedded search box from the GUI, the functionality does not work as expected.
Workaround:
None.
Component: Application Security Manager
Symptoms:
Changes in security policy are applied after ASM restart. This may activate unintended enforcement.
Conditions:
Restart ASM.
Impact:
Potentially unintended activation of new security entities.
Workaround:
None.
Component: Application Security Manager
Symptoms:
After ASM Signature update ASM-SignatureFile_20191117_112212.im is installed, a mistaken value for Attack Type (SSRF) appears incorrectly in REST query results.
Conditions:
ASM Signature update ASM-SignatureFile_20191117_112212.im is installed, even if another ASM Signature update is installed subsequently.
Impact:
A mistaken value for Attack Type (SSRF) appears incorrectly in REST query results. This impacts BIG-IQ usage and other REST clients.
Workaround:
Workaround:
1) Install a newer ASU to reassociate the affected signatures with the correct attack type
2) Run the following SQL on the affected BIG-IP devices:
DELETE FROM PLC.NEGSIG_ATTACK_TYPES WHERE attack_type_name = "Server-Side Request Forgery (SSRF)";
Component: TMOS
Symptoms:
If the system daemon responsible for LDAP authentication crashes, the system will not automatically restart it, and remote LDAP authentication may stop working.
In /var/log/daemon.log, you may see the following:
warning systemd[1]: nslcd.service failed
Conditions:
Nslcd daemon crashed, and it fails to restart.
Impact:
System authentication stops working until nslcd is restarted.
Workaround:
Manually restart nslcd daemon:
tmsh start sys service nslcd
Component: TMOS
Symptoms:
In rare circumstances, TMM may core when changing the ike-peer configuration from IKEv2 to IKEv1.
Conditions:
- The BIG-IP system is attempting to establish an IKEv2 tunnel.
- The related ike-peer config is changed from IKEv2 to IKEv1.
Impact:
Traffic disrupted while tmm restarts.
Workaround:
Do not reconfigure the ike-peer configuration while the related IPsec tunnel is attempting to establish.
Component: Access Policy Manager
Symptoms:
APMD daemon crashes.
Conditions:
When configuring 'REST API Security (Open API Spec)' application with Rate Limiting in Security.
Specific conditions required to reproduce this issue have not been identified.
Impact:
APMD crashes and restarts during configuration time. No service impact.
Workaround:
No workaround is needed because it is during configuration time and the system recovers by itself.
Component: TMOS
Symptoms:
IPsec-related tmm crash and generated core file during rekey.
Conditions:
-- IPsec timeout occurs.
-- Some temporary SA's are created by racoon.
Impact:
Tmm crashes and creates core file. Traffic disrupted while tmm restarts.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
Monitor continues to send previously configured settings from the server SSL profile such as client certificate or cipher list after the SSL profile has been removed/modified from the monitor.
Conditions:
-- In-TMM monitor configured.
-- SSL monitor configured with a server SSL profile.
-- Modification of the server SSL profile on the monitor.
Impact:
The previously configured settings, such as certificate or cipher, may continue to be transmitted to the server, resulting in node continuing to be marked up or down (respectively).
Workaround:
You can use either of the following workarounds:
-- Disable and then re-enable the pool member.
-- Restart tmm. Note: Using this option interrupts traffic. Traffic disrupted while tmm restarts.
Component: Application Security Manager
Symptoms:
When sending request to a protected API URL, with a trusted bot signature, tmm tries to perform reverse DNS to verify the signature. During this process, the URL qualification might change. In this case - tmm crashes.
Conditions:
-- Bot Defense profile attached.
-- 'API Access for Browsers and Mobile Applications' is enabled.
-- A DNS server is configured.
-- Request is sent to an API-qualified URL.
-- Request is sent with a trusted bot signature.
Impact:
Traffic disrupted while tmm restarts.
Workaround:
Disable the 'API Access for Browsers and Mobile Applications' or remove the DNS server.
Component: Application Security Manager
Symptoms:
Dosl7 remote logger messages breaks ArcSight CEF connector when using ArcSight destination format. CEF Logs are dropped.
Conditions:
- ASM provisioned
- Dos profile attached to a virtual server
- Dos application protection enabled
- Logging profile configured with ArcSight format attached to a virtual
Impact:
ArcSight server might be broken after getting dosl7 attack detection messages from the BIG-IP.
Workaround:
BIG-IP iRule or another proxy can be used to intercept ArcSight messages and strip the a string portion from ArcSight numeric type fields.
Component: TMOS
Symptoms:
Validation does not prevent specific configuration, but reports errors. In newer versions:
-- err tmm1[7019]: 01010008:3: Proxy initialization failed for /Common/vs_test. Defaulting to DENY.
-- err tmm1[7019]: 01010008:3: Listener config update failed for /Common/vs_test: ERR:ERR_ARG
In older versions:
-- err tmm[23118]: 01010007:3: Config error: virtual_server_profile no suitable hudchain
-- err tmm[23118]: 01010007:3: Config error: add virtual server profile error
Conditions:
Creating a virtual server with UDP, HTTP, SSL, (and OneConnect) profiles.
Impact:
Virtual server is defined and in configuration, but does not pass traffic.
On v12.1.x and v13.0.0, attempts to recover from this configuration can leave TMM in a bad state, which can then result in a TMM crash.
Workaround:
None.
Component: Global Traffic Manager (DNS)
Symptoms:
The output of a REST call to tm/gtm/wideip/<wideip_kind> returns objects that contain inconsistent values for the property "lastResortPool". For instance, for the kind "aaaa", the output might be:
"lastResortPool": "aaaa \"\""
Conditions:
The BIG-IP admin has modified a WideIP object via tmsh and used the following command structure:
"tmsh modify gtm wideip <wideip_kind> www.example.com last-resort-pool <pool_kind>"
Impact:
The lastResortValue in the REST response might be confusing for an external orchestrator that consumes the BIG-IP configuration via iControl REST. BIG-IQ, for instance, BIG-IQ might be confused with these values
Workaround:
Change the wideIP object via the GUI and set the Last Resort Pool to None, then save the changes.
Component: Application Security Manager
Symptoms:
After rebooting a vCMP guest with ASM provisioned and configured, there are no asm policies. The following command returns no results:
tmsh list asm policy all-properties
Conditions:
-- vCMP host with 2+ slots
-- vCMP guest with 2+ slots
-- LTM+ASM provisioned
-- ASM security policy + virtual server configured
-- reboot primary slot of VCMP host
Impact:
There are no ASM policies on the vCMP guest.
Component: Application Security Manager
Symptoms:
Incorrect access privileges to "Policy List" and "Security Policy Configuration" pages are given
Conditions:
User has a complex role, e.g. ASM Admin in partition A and Guest in partition B
Impact:
User with complex role gets incorrect privileges in "Policy List" and "Security Policy Configuration" pages
Workaround:
Component: Application Security Manager
Symptoms:
The 'Enforcement Mode' field in the security policy list has no value.
Conditions:
Security policy list in the GUI is sorted by 'last modified'.
Impact:
Security policy list page has empty 'Enforcement Mode' field.
Workaround:
None.
Component: TMOS
Symptoms:
The error Response body from iControl REST is truncated at 2048 characters. If an iControl REST response sends an error that is longer than 2048 characters, the truncated response will not contain valid JSON.
Conditions:
This occurs when iControl REST error messages are longer than 2048 characters.
Impact:
The error response body is deformed when the length of the error body is more than 2k characters
Component: TMOS
Symptoms:
After clicking UI Security :: Network Firewall : Active Rules, /var/log/ltm reports the following error message:
--warning postgres ERROR: syntax error at or near 'FROM' at character 17.
Conditions:
Enabled turboflex-security and AFM module.
Impact:
-- Possible leak of postgres database connections.
-- A warning log message is created, but the system continues to function normally.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
A QUIC connection does not complete the handshake successfully when the Client Hello spans multiple initial packets.
Conditions:
-- QUIC is in use.
-- A Client Hello is received that spans multiple packets.
Impact:
QUIC is unable to process a Client Hello that spans multiple packets.
Workaround:
None.
Component: Application Visibility and Reporting
Symptoms:
The incorrect goodput value is showing on the GUI > Analytics > TCP > Goodput.
Conditions:
AVR is provisioned
Running TCP related traffic (with the amount that can exceeds the MAX_INT value in any aggregation level).
Impact:
AVR statistics for TCP goodput may be incorrect.
Workaround:
There is no workaround at this time.
Component: TMOS
Symptoms:
When manual device group sync is enabled, the device with the most recent change will be displayed at the top of the Device Management::Overview GUI page.
Conditions:
-- Multiple devices in a device group
-- Device group has manual config sync enabled
-- A change is made on a device
Impact:
When the list loads, the device with the most recent changes is displayed at the top. This can make the device order appear to be inconsistent, and can create confusion when doing manual config sync if you are expecting the order to be always consistent.
Component: Application Security Manager
Symptoms:
After creating certain security policies, errors are logged to /var/log/asm:
crit g_server_rpc_handler_async.pl[2328]: 01310027:2: ASM subsystem error
(asm_config_server.pl,F5::DbUtils::insert_data_to_table): Row 7 of table PLC.PL_SUGGESTIONS is missing viol_index () -- skipping
Conditions:
This occurs when creating security policy based on the following security templates:
- drupal
- owa
- sharepoint
- wordpress
Impact:
Misleading errors are logged in ASM log file; they can be safely ignored.
Component: Local Traffic Manager
Symptoms:
The Global SNAT feature does not work with HTTP2.
Conditions:
-- Global SNAT is used
-- HTTP2 is used.
Impact:
Traffic uses the incorrect IP addresses when sourced from the BIG-IP system.
Workaround:
Use an explicit SNAT setting: SNAT Auto-Map or a SNAT pool.
Component: TMOS
Symptoms:
The 'SSL Profile (Client)' and 'SSL Profile (Server)' listboxes (both 'Selected' and 'Available') now have a fixed width when viewing Virtual Server settings.
Conditions:
-- An SSL profile (client or server) with a long name.
-- Accessing the Virtual Server settings page in the GUI.
Impact:
If many SSL profiles start with the same several letters, it may be impossible to detect which one is the desired profile.
Workaround:
None.
Component: TMOS
Symptoms:
Despite having free memory, the BIG-IP system logs kernel page allocation failures to the /var/log/kern.log file. The first line of the output appears similar to the following example:
swapper/13: page allocation failure: order:2, mode:0x204020
After that, a stack trace follows. The process name in the line ('swapper/16', in this example). You may see generic Linux processes or processes specific to F5 in that line.
Conditions:
This issue is known to occur on the appliance 10350V-F D112.
Impact:
As different processes can experience this issue, the system may behave unpredictably. For example, it is possible for a TMOS installation to fail as a result of this issue. Other processes may not exhibit any side effect as a result of this issue. The exact impact depends on which process becomes affected and how this process is designed to handle such a failure to allocate memory.
Workaround:
You can work around this issue by increasing the value of the min_free_kbytes kernel parameter. This controls the amount of memory that is kept free for use by special reserves.
It is recommend to increase this to 128 MB (131072 KB).
When instantiating this workaround, you must consider whether you want the workaround to survive only reboots, or to survive reboots, upgrades, RMAs, etc. This is an important consideration to make, as you should stop using this workaround when this issue is fixed in a future version of BIG-IP software. So consider the pros and cons of each approach before choosing one.
-- If you want the workaround to survive reboots only, perform the following procedure:
1) Log on to the advanced shell (BASH) of the primary blade of the affected system.
2) Run the following commands (with the desired amount in KB):
# clsh "sysctl -w vm.min_free_kbytes=131072"
# clsh "echo -e '\n# Workaround for ID 851785' >> /etc/sysctl.conf"
# clsh "echo 'vm.min_free_kbytes = 131072' >> /etc/sysctl.conf"
-- If you want the workaround to survive reboots, upgrades, RMAs, etc., perform the following procedure:
1) Log on to the advanced shell (BASH) of the primary blade of the affected system.
2) Run the following commands (with the desired amount in KB):
# clsh "sysctl -w vm.min_free_kbytes=131072"
# echo -e '\n# Workaround for ID851785' >> /config/startup
# echo 'sysctl -w vm.min_free_kbytes=131072' >> /config/startup
Note that the last two commands are not wrapped inside 'clsh' because the /config/startup file is already automatically synchronized across all blades.
Once the issue is fixed in a future BIG-IP version, remove the workarounds:
-- To remove the first workaround:
1) Edit the /etc/sysctl.conf file on the BIG-IP appliance and remove the added lines at the bottom.
2) Reboot the system by running 'clsh reboot'. This will restore the min_free_kbytes kernel parameter to its default value for the BIG-IP version you are running.
-- To remove the second workaround:
1) Edit the /config/startup file on the BIG-IP appliance and remove the extra lines at the bottom.
2) Reboot the system by running 'clsh reboot'. This restores the min_free_kbytes kernel parameter to its default value for the BIG-IP version you are running.
To verify the workaround is in place, run the following command (this should return the desired amount in KB):
# clsh "cat /proc/sys/vm/min_free_kbytes"
Component: Local Traffic Manager
Symptoms:
When a TLS END_OF_EARLY_DATA message is received by QUIC, a CONNECTION_CLOSE frame with a CRYPTO_ERROR is produced. However, the error should be a PROTOCOL_VIOLATION.
Conditions:
-- QUIC is in use.
-- A TLS END_OF_EARLY_DATA message is received.
Impact:
A CRYPTO_ERROR is produced, however, the error should be PROTOCOL_VIOLATION.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
The BIG-IP QLOG implementation is currently on draft-00, and draft-01 has been released.
Conditions:
QUIC and QLOG are in use.
Impact:
Existing third-party applications might remove support for QLOG draft-00 at some point.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
When blades 1 and 4 are disabled on the active chassis, the failover period is between 3.4 to 4.7 seconds before the next-active device starts processing messages.
If the blades are physically pulled from the chassis,
the failure occurs within 1 second.
Conditions:
-- Multi-blade VIPRION system
-- Blades 1 and 4 are connected to the network via trunks, blades 2 and 3 are CPU-only blades
-- Blades 1 and 4 are disabled via the GUI
Impact:
Significant delay before BIG-IP delivers a web page during between-cluster failover
Component: Local Traffic Manager
Symptoms:
Invalid pseudo header or malformed header in an HTTP/3 request should result in resetting of the stream with error code of HTTP3_GENERAL_PROTOCOL_ERROR. Instead the connection is reset with HTTP3_UNEXPECTED_FRAME error code.
Conditions:
-- Virtual server with QUIC, HTTP/3, HTTP, SSL, and httprouter profiles.
-- HTTP/3 header frame from client with invalid or malformed header is received.
Impact:
Connection is reset with incorrect error code, instead of just the individual stream.
Workaround:
No workaround.
Component: Local Traffic Manager
Symptoms:
Debug logging in the database monitor daemon (DBDaemon) for database health monitors (mssql, mysql, postrgresql, oracle) is enabled on a per-monitor basis.
When a ping is initiated for a particular monitor with debug logging enabled in the monitor configuration, debug logging in DBDaemon is enabled.
When a ping is initiated for a particular monitor with debug logging disabled in the monitor configuration, debug logging in DBDaemon is disabled.
When monitoring database pool members with a mix of monitors with debug logging enabled vs. disabled, the result can be that debug logging in DBDaemon is enabled and disabled at times which do not correspond to all actions related to a specific database monitor, or pool members monitored by that monitor.
In addition, debug messages logging internal DBDaemon state related to the management of the full collection of monitored objects, active threads, etc. may not be logged consistently.
Conditions:
-- Using multiple database health monitors (mssql, mysql, postrgresql, oracle)
-- Enabling debug logging on one or more database health monitors, but not all
Debug logging for database health monitors is enabled by configuring the "debug" property of the monitor with a value of "yes".
Debug logging is disabled by configuring the "debug" property with a value of "no" (default).
# tmsh list ltm monitor mysql mysql_example debug
ltm monitor mysql mysql_example {
debug yes
Impact:
Logging of database monitor activities by DBDaemon may be inconsistent and incomplete, impeding efforts to diagnose issues related to database health monitors.
Workaround:
When attempting to diagnose database health monitor issues with DBDaemon debug logging, enable debug logging for ALL database monitors currently in use.
Once diagnostic data collection is completed, disable debug logging for all database monitors currently configured/in use.
Component: Local Traffic Manager
Symptoms:
Unable to establish active FTP connection with custom FTP filter.
Conditions:
All of the following conditions are true:
-- Virtual server using custom FTP filter.
-- FTP filter has port (port used for data channel) set to 0 (zero).
-- Virtual server has source-port set to preserve-strict.
-- Using active FTP through the virtual server.
Impact:
-- The active FTP data channel is reset.
-- Commands that require data channel in active mode fail.
Workaround:
-- Change source-port to change or preserve.
-- Set port on FTP filter to be used for data channel.
-- Use passive FTP.
Component: TMOS
Symptoms:
TMSH error example:
Configuration error: Can't associate ASM device sync (/Common/testsync/staging.example.com) folder does not exist
Conditions:
The conditions under which this occurs are unknown.
Impact:
Load of config file fails with an error that the folder does not exist.
Workaround:
Use 'tmsh load sys config verify', without specifying a specific file.
Component: TMOS
Symptoms:
The SNMPD daemon no longer shows up in the list of daemons on the high availability (HA) Fail-safe GUI page.
Conditions:
Viewing the page at:
System :: High Availability : Fail-safe : System
Impact:
Unable to configure the high availability (HA) settings for the snmpd high availability (HA) daemon through the GUI.
Workaround:
Use TMSH to modify the snmpd high availability (HA) settings.
Component: Application Security Manager
Symptoms:
Policies with characters that are encoded with urlencode in name cannot be exported in GUI
Conditions:
Policies has characters that are encoded with urlencode in name
Impact:
Policy cannot be exported in GUI
Workaround:
Most policies can be cloned, and during clone user can select name without special characters. Then cloned policy can be exported.
Component: Local Traffic Manager
Symptoms:
Incorrect MAC is seen in the network when virtual wire is configured with a FastL4 profile.
Conditions:
Virtual wire is configured with a FastL4 profile.
Impact:
Incorrect MAC on the packets.
Workaround:
None.
Component: Advanced Firewall Manager
Symptoms:
When attaching dos profile to virtual server, it will crash occasionally.
Conditions:
-- AFM is provisioned
-- Performing profile attachment
Impact:
Autodosd crashes and restarts.
Component: Application Security Manager
Symptoms:
Web app flow might fail resulting in JavaScript errors related to CSP policy
Conditions:
-- ASM provisioned.
-- Bot-Defense or DoS Application profile assigned to a virtual server.
-- The backend server sends CSP headers.
Impact:
Web application flow might fail.
Workaround:
Attach an iRule:
when HTTP_REQUEST {
set csp 0
when HTTP_RESPONSE {
if { [HTTP::header exists Content-Security-Policy] } {
set csp "[HTTP::header values Content-Security-Policy]"
when HTTP_RESPONSE_RELEASE {
if { $csp != 0 } {
HTTP::header replace Content-Security-Policy $csp
set csp 0
Component: Application Security Manager
Symptoms:
After you visit the Policy Inheritance page, there is high CPU usage in the browser until you leave the policies page.
Conditions:
This occurs when opening the Policy Inheritance page for a policy that does not have a parent.
Impact:
High CPU usage by browser
Workaround:
Component: TMOS
Symptoms:
The outgoing SCTP connection does not expire after attempting to INIT the maximum number of times. It then becomes stuck and does not expire when it reaches its idle-timeout, and cannot be manually deleted.
Conditions:
An outgoing SCTP connection is permitted to attempt the INIT retransmit the maximum number of times configured with no responses (accepting or aborting) from the target endpoint.
Impact:
Stale SCTP connections are left in the system and start to use up memory. Traffic may be interrupted in certain configurations, as the system thinks it is still attempting to bring up the lost SCTP connection and does not ever try to create a new one.
Workaround:
To clear the stale connections, restart tmm:
bigstart restart tmm
Note: Restarting tmm causes an interruption to traffic.
Component: Application Security Manager
Symptoms:
Importing a json policy to a BIG-IP that is in a device group machine causes errors on the other devices, which leads to a sync failure.
Conditions:
This can occur when a json policy is imported to a BIG-IP that is in a device group
Impact:
Config sync fails.
Component: Application Security Manager
Symptoms:
Link between 'API protection profile' and 'Security Policy' created with swagger based 'API protection profile' preserved in UCS file. This link is not restored after UCS upload.
Conditions:
UCS upload.
Impact:
'API protection profile' has no link to related security policy.
Workaround:
None.
Component: TMOS
Symptoms:
When the LCD is disabled or turned off on a VIPRION system, the blade status lights turn amber.
Conditions:
You can cause this to occur by running the command:
tmsh modify sys db platform.chassis.lcd value disable
Impact:
Blade status lights change to amber, even if nothing is wrong with the system.
Workaround:
None.
Component: Access Policy Manager
Symptoms:
Bad response, or invalid response for request in which default port was used in web-application.
Conditions:
Default port is used in the url, back end application does not expect the default port in the request's Host header.
Impact:
Web-application misfunction
Workaround:
Custom workaround iRule can be used to remove default port form rewritten url.
Component: Access Policy Manager
Symptoms:
Policy exported and then imported with Syntax Error:
Import error: Syntax Error:(/shared/tmp/apmom/import/p--11-02--07-34-31--208.conf at line: 976) "src-subnet" unexpected argument.
Conditions:
Elements in policy are very large (e.g., 100 KB).
Note: This is a rarely occurring issue.
Impact:
Unable to import policy.
Workaround:
None.
Component: Global Traffic Manager (DNS)
Symptoms:
The bigip_gtm.conf is reverted to default after rebooting (or upgrading to a newer BIG-IP software release).
Conditions:
-- The BIG-IP license is expired prior to the reboot or upgrade.
-- GTM is configured.
Impact:
The GTM configuration (in /config/bigip_gtm.conf) information is lost in the newly installed boot location.
Workaround:
Renew license before reboot. Always reboot with valid license.
If you have already rebooted or upgraded with an expired license, and your configuration has been lost, you can restore it using the following procedure.
1. Re-activate the BIG-IP license
2. Restore bigip_gtm.conf from the auto-created backup (.bak) file:
cp /config/bigip_gtm.conf.bak /config/bigip_gtm.conf
3. Load the replaced config:
tmsh load sys config gtm-only
If this is a the result of a software upgrade, and the .bak file is not available or has been overwritten, you can boot back to the previous volume and re-copy the configuration from there (cpcfg or via the GUI) before rebooting back to the upgraded software release.
Component: Local Traffic Manager
Symptoms:
Validation for TCP::collect was changed in BIG-IP software v12.0.0 (with the introduction of JET specifications). Prior to 12.0.0, there were no restrictions on the values of the two arguments. As of 12.0.0, the first argument ('collect_bytes') must be a positive integer, and the second argument ('skip_bytes) must be a non-negative integer.
Occurrences of 'TCP::collect 0 0' in iRules experience issues when upgrading to a newer version, producing warnings in LTM log:
/Common/T_collect:9: warning: [The following errors were not caught before. Please correct the script in order to avoid future disruption. "invalid argument 0; expected syntax spec:"136 17][TCP::collect 0 0].
Conditions:
-- Using a version of BIG-IP software earlier than 12.0.0, configure an iRule with a 'TCP::collect 0 0' command.
-- Upgrade to 12.0.0 or later.
Impact:
Warning in the LTM log file. The iRules containing 0 values do not function as expected. There is no other impact.
Workaround:
Change 'TCP::collect 0 0' to a value other than 0 (zero) in any iRules before or after upgrade.
Component: Local Traffic Manager
Symptoms:
Traffic fails to pass through a virtual server.
Conditions:
-- Virtual server is removed and a new one is added in a single transaction.
-- Virtual server references a plugin profile.
For example, create a CLI transaction:
- delete ltm virtual vs_http
- create ltm virtual vs_https destination 1.1.1.1:443 vlans-enabled profiles replace-all-with { http ntlm oneconnect }
- submit cli transaction
Impact:
Traffic failure on the new virtual server.
Workaround:
Create a virtual server that does not accept any traffic, but keeps the NTLM MPI plugin channel alive:
tmsh create ltm virtual workaround destination 1.1.1.1:1 profiles replace-all-with { http oneconnect ntlm } vlans-enabled vlans none && tmsh save sys config
Component: TMOS
Symptoms:
Config script does not refresh management address entry properly when alternating between dynamic (DHCP) and static configuration.
Conditions:
- Management IP assignment is changed from dynamic (DHCP) to static.
- Same IP address is configured, as previously received from DHCP server.
Impact:
Remote management access is lost after DHCP lease expires.
Workaround:
Restart BIG-IP after changing the management IP address.
Component: Local Traffic Manager
Symptoms:
Translucent vlan-groups may set the locally-unique bit in a destination MAC address when sending traffic to a pool member/client.
Conditions:
On versions earlier than 15.0.0:
- Translucent vlan-group is in use.
On v15.0.0 and later:
-- Translucent vlan-group is in use.
-- The connection.vgl2transparent db variable is enabled.
Impact:
Traffic handled by translucent vlan-groups may not work properly.
Workaround:
On versions earlier than 15.0.0, there is no workaround.
-- On version 15.0.0 and later, you can disable the connection.vgl2transparent db variable to mitigate the problem:
tmsh modify sys db connection.vgl2transparent value disable
Note: connection.vgl2transparent is disabled by default.
Component: Application Security Manager
Symptoms:
Learning suggestions created from single request do not show source 'request log' in the 'Suggestion' GUI section.
Conditions:
'Learning Suggestion' created from only one 'Request Log' record.
Impact:
Learning suggestions created from single request does not show source 'request log' in the 'Suggestion' GUI section
Workaround:
None.
Component: TMOS
Symptoms:
Rest API returns 404 'Object not found"' error when attempting direct access to pool member that has pipe symbol '|' in the server or virtual server name.
Conditions:
An iControl/REST call to a pool member that has a virtual server on the Server whose name contains a | character in the server or virtual server name.
Impact:
The iControl/REST call cannot manage a pool member associated with a virtual server or server whose name contains a | character.
Workaround:
Rename the server or virtual server to a name that does not contains the | character.
Component: Application Security Manager
Symptoms:
UCS backup archive file size is much larger than UCS files in previous releases.
Conditions:
-- UCS backup process finishes with failure and does not clean temporary files.
-- A second UCS backup is attempted.
Impact:
Those files are included in the UCS archive, which results in an unusually large UCS backup files.
Workaround:
Before running the UCS backup process, remove directories:
/var/tmp/ts_db.save_dir_*.cstmp/
Component: Application Security Manager
Symptoms:
After you update the swagger file of a policy, some parameters that are not defined in the updated swagger may remain in the policy.
Conditions:
1. Policy contains global parameters that were added manually
2. All the URL parameters are deleted in the new swagger file
Impact:
Traffic to these parameters will not raise a violation ILLEGAL PARAMETER as expected
Workaround:
The leftover parameters need to be removed manually
Component: Local Traffic Manager
Symptoms:
This release includes a new base client-ssl profile for use by QUIC virtual servers. If an existing client-ssl profile named 'clientssl-quic' exists, it will be overwritten by the new built-in profile after upgrading.
Conditions:
The system to be upgraded has an existing client-ssl profile named 'clientssl-quic'.
Impact:
The existing profile will be overwritten by the new built-in profile.
Workaround:
Rename the existing profile prior to upgrade.
Component: Local Traffic Manager
Symptoms:
If you try to assign an iRule to a Transport Config, and if the iRule has a proc that references a datagroup, the assignment fails with an error:
01070151:3: Rule [/Common/test2] error: Unable to find value_list (datagroup) referenced at line 6: [class lookup "first" datagroup]
Conditions:
-- Assign an iRule to a Transport Config.
-- The iRule has a proc.
-- The proc references a datagroup.
Impact:
Validation fails. An iRule with a proc referencing a datagroup cannot be assigned to Transport Config objects.
Workaround:
Make the datagroup a Tcl variable to bypass validation.
Component: Policy Enforcement Manager
Symptoms:
Tmm crashes.
Conditions:
-- BIG-IP PEM is licensed and configured.
-- Heavy traffic is received by PEM virtual server.
-- The traffic pattern goes through subscriber add/delete frequently.
Impact:
Traffic disrupted while tmm restarts.
Workaround:
None.
Component: TMOS
Symptoms:
The 'tmsh save /sys config' command hangs and fails to save the configuration if there is a memory allocation failure when creating the reply.
Conditions:
-- A large number of iApps: in the thousands.
-- Each iApp has tens of variables.
Impact:
Because tmsh cannot save the configuration, if the BIG-IP system reboots, any changes made since the last successful save are lost.
Workaround:
Run the command:
tmsh save /sys config binary
This does not save the configuration to files in /config, but it does at least allow you to save the binary configuration.
That way, you can reboot the BIG-IP system and not lose the configuration.
Note: It is possible that a reboot will provide sufficient memory to save to configuration files. It depends on the configuration of virtual memory at the time of the save. It is possible that every time you want to save the config, you must use the binary option.
Component: Advanced Firewall Manager
Symptoms:
AVR analytics is reporting null domain name for a DNS query if DNS DoS profile is attached to a virtual server, but the profile does not have the matching type vector enabled to the query type.
Conditions:
-- DNS DoS profile is attached to a virtual server.
-- The query type in the DNS query does not match an enabled DNS vector on the DNS profile.
Impact:
DNS domain name is reported as NULL
Workaround:
Enable the matching type vector on the DNS DoS profile.
Component: Application Security Manager
Symptoms:
One of the suggestion details is placed incorrectly, out of alighment.
Conditions:
This occurs when you open the details for learning suggestion, e.g., based on refinement.
Impact:
Refinement title is out of line.
Workaround:
Use a different browser, if needed.
Component: Local Traffic Manager
Symptoms:
Because of the Tcl error, connection gets reset and reports an error:
err tmm[18774]: 01220001:3: TCL error: /Common/test2- bad port in node <addr> <port> cmdTCL error (line 43) (line 43) invoked from within "node 172.x.x.x IP [LB::server port]"
Conditions:
Using node command under pre-load-balancing iRule events.
Impact:
Unclear port values in Tcl error message.
Workaround:
None.
Component: Service Provider
Symptoms:
The tmsh context-sensitive help content for the following diameter session attributes is missing:
-- respond-unroutable
-- retransmission-action
-- retransmission-queue-limit-high
-- retransmission-queue-limit-low
-- retransmission-queue-max-bytes
-- retransmission-queue-max-messages
Conditions:
When attempting in tmsh to list a diameter session profile followed by a question mark for context-sensitive help- for example:
list ltm message-routing diameter profile session <sess-name> ?
Impact:
The specified attributes are no described.
Workaround:
These are the missing descriptions:
-- respond-unroutable: When selected (enabled), messages that do not match any known route will be transformed into an error answer message and sent to the originator of the request. When disabled, unroutable request messages are routed back to the connection where they came from. The default value is disabled.
-- retransmission-action: Specifies the action performed when retransmission has been triggered for a request message. The options are:
1) Disabled: Retransmission is disabled. This is the default action.
2) Busy: An answer message is generated with a TOO_BUSY result code and returned to the originator of the request.
3) Unable: An answer message is generated with an UNABLE_TO_DELIVER result code and returned to the originator of the request.
4) Retransmit: The request message will be retransmitted.
-- retransmission-queue-limit-high: Specifies the high watermark for the retransmission queue (in percentage). If the retransmission queue exceeds this limit, the transport window will begin closing. A value of 0 will disable closing the transport window. Valid range from 0 to 100. The default value is 90.
-- retransmission-queue-limit-low: Specifies the low watermark for the retransmission queue (in percentage). If the retransmission queue drops below this limit, the transport window will reopen. Valid range from 0 to 100. The default value is 60.
-- retransmission-queue-max-bytes: Specifies the maximum number of bytes that can be stored in a connections retransmission queue. A value of 0 will disable this limit. The default value is 131072 bytes.
-- retransmission-queue-max-messages: Specifies the maximum number of messages that can be stored in a connections retransmission queue. A value of 0 will disable this limit. The default value is 1024 messages.
Component: TMOS
Symptoms:
TMSH currently allows you to specify the 'add-on-keys' option when running the 'revoke sys license' command, but the option is not honored and the entire license is revoked.
Conditions:
-- BIG-IP license and add-on license are installed.
-- Attempt to revoke the system license with 'add-on-keys' as an option.
Impact:
Add-on-keys option is ignored, and the entire license is revoked instead.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
Files imported from iRules LX workspace may have incorrect SELinux contexts such as abrt_var_cache_t.
This can cause reloading the workspace to fail with errors:
01070079: failed to create workspace archive ... Return code {2}
Conditions:
Importing the iRules LX workspace.
Impact:
Workspace cannot be imported
Workaround:
As a workaround you can run the following command on the folders to restore the context:
restorecon -R -v
Component: TMOS
Symptoms:
Systemd-journald cannot handle logs with embedded newlines, write trailing content to /var/log/user.log. Bare ')' being logged to /var/log/user.log., for example:
cat /var/log/user.log
[...]: Deleting file /shared/core/*.core.*
[...]: Deleting file /shared/core/*.core.*
[...] )
[...] )
Conditions:
This occurs when the system logs syslog messages containing embedded newlines, such as
- The cron process tries and fails to send an email because of output about a cron script.
- Modify syslog include configuration
- Apply ASM policy configuration change
- GTM.debugprobelogging output from big3d
Impact:
The logging subsystem accepts syslog messages with embedded newlines, and writes first line to the appropriate file, and remaining lines to /var/log/user.log.
Workaround:
View the logs using journalctl -D /var/log/journal
Component: Local Traffic Manager
Symptoms:
SSL handshake fails with error in LTM logs.pkcs11d[10407]:
err pkcs11d[10407]: 01680048:3: C_Sign: pkcs11_rv=0x00000082, CKR_OBJECT_HANDLE_INVALID
Conditions:
Key created with Safenet NetHSM is used in SSL profile for virtual server. This error is seen randomly.
Impact:
SSL handshake fails.
Workaround:
Restart the PKCS11D.
Component: Local Traffic Manager
Symptoms:
When the conditions are met, if the interface of the connection on the active system changes, the peer does not get notified of this, and that connection persists on the standby system even after the connection on the active system has been destroyed.
Conditions:
-- Using mirrored connections in a DSC.
-- Not using auto-lasthop with mirrored connections.
-- VLAN-keyed connections are enabled.
Impact:
Leaking connections on the standby system.
Workaround:
You can use either of the following workarounds:
-- Use auto-lasthop with mirrored connections.
-- Depending on the BIG-IP system's configuration, disabling VLAN-keyed connections may resolve this.
Component: Application Security Manager
Symptoms:
If there are trusted IP addresses in the selected template, they are not shown in GUI during policy creation
Conditions:
Create user-defined template from policy with trusted IP addresses.
Impact:
If you manually enter the same IP addresses that were in template, you may get an error message after policy creation
Workaround:
None.
Component: iApp Technology
Symptoms:
When the iApp, f5.automated_backup, script is terminated due to the max-script-run-time, the script still continues and finishes, sometimes with scriptd coring and posting error messages in /var/log/ltm:
-- info logger[17173]: f5.automated_backup iApp autobackup: STARTED
-- info logger[17175]: f5.automated_backup iApp autobackup: pem.f5lab.com_20191004.ucs GENERATING
-- err scriptd[13532]: 014f0004:3: script has exceeded its time to live, terminating the script <------ after 20 secs, it continues even after the scriptd core.
-- notice sod[3235]: 01140041:5: Killing /usr/bin/scriptd pid 13532.
-- warning sod[3235]: 01140029:4: high availability (HA) daemon_heartbeat scriptd fails action is restart.
-- info logger[19370]: f5.automated_backup iApp autobackup: pem.f5lab.com_20191004.ucs SAVED LOCALLY
(/var/local/ucs)
-- info logger[19372]: f5.automated_backup iApp autobackup: FINISHED
Conditions:
Configure the iApp application with f5.automated_backup template to do auto-backup at regular intervals.
Impact:
Scriptd core.
Workaround:
Increasing the sys scriptd max-script-run-time higher then the default of 300 seconds might be helpful if the higher timeout allows the script to finish
For example, if the script is saving a UCS and the save takes 400 seconds, then increasing the max-script-run-time to 430 seconds would allow the script to finish and would workaround this issue.
Component: TMOS
Symptoms:
When a BIG-IP instance goes offline, any functioning tunnel is removed from the active configuration. Upon restoration to online operation, the tunnel is not automatically restored.
Conditions:
-- Configuration includes tunnels.
-- BIG-IP instance goes offline and then comes back online.
Impact:
Failure of tunnel packet traffic.
Workaround:
Manually recreate the tunnel after the BIG-IP instance has been brought back online.
Component: Access Policy Manager
Symptoms:
You are unable to configure Verified Accept on SSL Orchestrator.
Conditions:
-- SSL Orchestrator in use .
-- A TCP profile is in use and it contains the Verified Accept flag.
Impact:
No connectivity over SSL Orchestrator.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
When FIPS mode is set to use FIPS 140-2 Level 3 protection, new keys cannot be created in the module's protected partition
Conditions:
-- FIPS 140-2 Level 3 protection is configured on a NetHSM partition
-- You attempt to create a FIPS key using that partition
Impact:
New Keys cannot be created
Workaround:
Here are all the steps to generate a new netHSM key called "workaround" and install it into the BIG-IP config:
[root@bigip1::Active:Standalone] config # fipskey.nethsm --genkey -o workaround -c module
WARNING: fipskey.nethsm will soon be deprecated for use with Thales. Please switch to using tmsh commands instead.
tmsh commands...
Generate Key:
tmsh create sys crypto key <key_name> security-type nethsm [gen-certificate|gen-csr] ...
For an exhaustive list of options, please consult F5's tmsh documentation.
Generate CSR for existing key:
tmsh create sys crypto csr <csr_name> key <key name> ...
For an exhaustive list of options, please consult F5's tmsh documentation.
Generate Self-Signed Certificate for existing key:
tmsh create sys crypto cert <cert_name> key <key name> ...
For an exhaustive list of options, please consult F5's tmsh documentation.
Delete Key:
tmsh delete sys crypto key <keyname>
str[cd /shared/tmp && /opt/nfast/bin/generatekey -b pkcs11 certreq=yes selfcert=yes protect=module size=2048 embedsavefile="workaround" plainname="workaround" digest=sha256]
key generation parameters:
operation Operation to perform generate
application Application pkcs11
protect Protected by module
verify Verify security of key yes
type Key type RSA
size Key size 2048
pubexp Public exponent for RSA key (hex)
embedsavefile Filename to write key to workaround
plainname Key name workaround
x509country Country code
x509province State or province
x509locality City or locality
x509org Organisation
x509orgunit Organisation unit
x509dnscommon Domain name
x509email Email address
nvram Blob in NVRAM (needs ACS) no
digest Digest to sign cert req with sha256
Key successfully generated.
Path to key: /opt/nfast/kmdata/local/key_pkcs11_ua882aa9fadee7e440772cb6686358f4b283922622
Starting synchronisation, task ID 5de83486.6e9e32d7f367eaf4
Directory listing failed: No such file or directory
2. (this is to confirm the key is present with the label "workaround"
[root@bigip1::Active:Standalone] config # nfkminfo -l
Keys with module protection:
key_pkcs11_ua882aa9fadee7e440772cb6686358f4b283922622 `workaround'
Keys protected by cardsets:
[root@bigip1::Active:Standalone] config # tmsh install sys crypto key workaround from-nethsm
4. (install public certificate)
[root@bigip1::Active:Standalone] config # tmsh install sys crypto cert workaround from-local-file /config/ssl/ssl.crt/workaround
Component: Application Security Manager
Symptoms:
The GUI becomes unresponsive when you perform an operation that takes a long time (e.g., Attack Signatures update).
Conditions:
Long-running task is performed, such as export/import/update signatures.
Impact:
GUI is unresponsive for that session.
Workaround:
If you need to continue working during long task is performed, you can log in via another browser.
Component: TMOS
Symptoms:
The dynamic BWC policy can be attached from iRules but not detached. No error occurs when BWC::policy detach is run, but the detached policy continues to work.
Conditions:
-- Dynamic BWC policy for a HTTP request URI during session.
-- Running BWC::policy detach.
Impact:
The detached policy continues to work.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
Blades in a VIPRION system connect with one another over a data backplane and a management backplane.
For more information on the manner in which blades interconnect over the data backplane, please refer to K13306: Overview of the manner in which the VIPRION chassis and blades interconnect :: https://support.f5.com/csp/article/K13306.
Should an internal interface fail and thus block communication over the data backplane between two distinct blades, an unusual situation arises where different blades compute different CMP states.
For example, if on a 4-slot chassis, blades 2 and 3 become disconnected with one another, the following is TMM's computation of which slots are on-line:
slot1: slots 1, 2, 3, and 4 on-line (cmp state 0xf / 15)
slot2: slots 1, 2, and 4 on-line (cmp state 0xb / 11)
slot3: slots 1, 3, and 4 on-line (cmp state 0xd / 13)
slot4: slots 1, 2, 3, and 4 on-line (cmp state 0xf / 15)
As different slots are effectively operating under different assumptions of the state of the cluster, application traffic does not flow as expected. Some connections time out or are reset.
You can run the following command to inspect the CMP state of each slot:
clsh 'tmctl -d blade -s cmp_state tmm/cmp'
All slots should report the same state, for instance:
# clsh 'tmctl -d blade -s cmp_state tmm/cmp'
=== slot 2 addr 127.3.0.2 color green ===
cmp_state
---------
=== slot 3 addr 127.3.0.3 color green ===
cmp_state
---------
=== slot 4 addr 127.3.0.4 color green ===
cmp_state
---------
=== slot 1 addr 127.3.0.1 color green ===
cmp_state
---------
When this issue occurs, logs similar to the following example can be expected in the /var/log/ltm file:
-- info bcm56xxd[4276]: 012c0015:6: Link: 2/5.3 is DOWN
-- info bcm56xxd[4296]: 012c0015:6: Link: 3/5.1 is DOWN
-- info bcm56xxd[4296]: 012c0012:6: Trunk default member mod 13 port 0 slot 2; CMP state changed from 0xf to 0xd
-- info bcm56xxd[4339]: 012c0012:6: Trunk default member mod 13 port 0 slot 2; CMP state changed from 0xf to 0xd
-- info bcm56xxd[4214]: 012c0012:6: Trunk default member mod 13 port 0 slot 2; CMP state changed from 0xf to 0xd
And a CMP transition will be visible in the /var/log/tmm file similar to the following example:
-- notice CDP: PG 2 timed out
-- notice CDP: New pending state 0f -> 0b
-- notice Immediately transitioning dissaggregator to state 0xb
-- notice cmp state: 0xb
For more information on troubleshooting VIPRION backplane hardware issues, please refer to K14764: Troubleshooting possible hardware issues on the VIPRION backplane :: https://support.f5.com/csp/article/K14764.
Conditions:
This issue arises after a very specific type of hardware failure. The condition is very unlikely to occur and is impossible to predict in advance.
Impact:
Application traffic is impacted and fails sporadically due to a mismatch in CMP states between the blades. Failures are likely to manifest as timeouts or resets from the BIG-IP system.
Workaround:
F5 recommends the following to minimize the impact of this potential issue:
1) For all highly available configurations (e.g., A/S, A/A, A/A/S, etc.).
The BIG-IP system has functionality, in all software versions, to enact a fast failover when the conditions described occur.
To ensure this functionality will trigger, the following configuration requirements must be met:
a) The mirroring strategy must be set to 'between'.
b) A mirroring channel to the next-active unit must be up.
c) The min-up-members option must be set to the number of blades in the chassis (e.g., 4 if there are 4 blades in the chassis).
Note: It is not required to actually configure connection mirroring on any virtual server; simply choosing the aforementioned strategy and ensuring a channel is up to the next-active unit will suffice. However, note that some configurations will benefit by also configuring connection mirroring on some virtual servers, as that can greatly reduce the number of affected connections during a failover.
2) For 'regular' standalone units.
If a VIPRION system is truly standalone (no kind of redundancy whatsoever), there is no applicable failsafe action, as you will want to keep that chassis online even if some traffic is impaired. Ensure suitable monitoring of the system is in place (e.g., remote syslog servers, SNMP traps, etc.), so that a BIG-IP Administrator can react quickly in the unlikely event this issue does occur.
3) For a standalone chassis which belongs to a pool on an upstream load-balancer.
If the virtual servers of a standalone VIPRION system are pool members on an upstream load-balancer, it makes sense for the virtual servers to report unavailable (e.g., by resetting all new connection attempts) so that the upstream load-balancer can select different pool members.
An Engineering Hotfix can be provided which introduces an enhancement for this particular use-case. A new DB key is made available under the Engineering Hotfix: tmm.cdp.requirematchingstates, which takes values 'enable' and 'disable'.
The default is 'disable', which makes the VIPRION system behave as in versions without the enhancement. When set to 'enable', the VIPRION system attempts to detect this failure and, if it does, resets all new connections. This should trigger some monitor failures on the upstream load-balancer and allow it to select different pool members.
Please note you should only request the Engineering Hotfix and enable this DB key when this specific use-case applies: a standalone VIPRION system which belongs to a pool on an upstream load-balancer.
When the new feature is enabled, the following log messages in the /var/log/ltm file indicate when this begins and stops triggering:
-- crit tmm[13733]: 01010366:2: CMP state discrepancy between blades detected, forcing maintenance mode. Unable to relinquish maintenance mode until event clears or feature (tmm.cdp.requirematchingstates) is disabled.
-- crit tmm[13262]: 01010367:2: CMP state discrepancy between blades cleared or feature (tmm.cdp.requirematchingstates) disabled, relinquishing maintenance mode.
Component: Local Traffic Manager
Symptoms:
LTM HTTP monitor GUI displays incorrect green status when related pool is down.
TMSH shows correct information
Conditions:
LTM HTTP monitor destination port does not match with pool member port.
Impact:
LTM HTTP marks the node down, but the Instances tab of the monitor in the GUI reports the status as green
Workaround:
You can use either of the following workarounds:
-- Use TMSH to get correct info.
-- Ensure that LTM HTTP monitor destination port does match pool member port.
Component: Local Traffic Manager
Symptoms:
Virtual servers do not forward any traffic but the SNAT does.
Conditions:
-- Multiple wildcard IP forwarding virtual servers with the same destination address.
-- SNAT is configured.
Impact:
IP forwarding virtual server does not pick up any traffic.
Workaround:
Delete and then re-create virtual servers.
Component: Application Security Manager
Symptoms:
The word "Applying" is displayed long after the policy has been applied.
Conditions:
This can occur when applying policies.
Impact:
It appears that Apply policy is stuck, when it is not.
Workaround:
Refresh the page, or look for the log entry in /var/log/asm
ASMConfig change: Apply Policy Task Apply Policy Task [update]: Status was set to COMPLETED.
Component: TMOS
Symptoms:
After following F5-recommended procedures for hot-swapping the left annunciator card on a C4800 chassis and replacing the top bezel, the LCD screen fails to load.
Conditions:
- C4800 chassis with 2 annunciator cards.
- Hot-swap the left annunciator card and replace the top bezel.
Impact:
-- Status light on the top bezel turns amber.
-- LCD becomes unresponsive, and continuously displays 'F5 Networks Loading...'.
Workaround:
1. Run the command:
tmsh modify sys db platform.chassis.lcd value disable
2. Wait 10 seconds.
3. Run the command:
tmsh modify sys db platform.chassis.lcd value enable.
This forces the LCD to sync back up with the VIPRION system and returns it to normal operation. The top bezel status light should turn green.
Component: Local Traffic Manager
Symptoms:
The documented examples for REST::send refers to REST endpoints that are not valid.
Conditions:
Viewing the documentation at https://clouddocs.f5.com/api/irules/REST__send.html.
Impact:
Invalid examples lead to potential confusion.
Workaround:
Use valid REST endpoints, documented at https://clouddocs.f5.com/api/icontrol-rest/APIRef.html.
Component: TMOS
Symptoms:
When a 'net ipsec ike-peer' object has the version attribute with more than one value, upgrading to version 15.1.0 results in a failed upgrade.
Conditions:
The version attribute has two values, in this example, 'v1' and 'v2.'
net ipsec ike-peer test {
my-cert-file default.crt
my-cert-key-file default.key
my-id-value 38.38.38.64
peers-id-value 38.38.38.38
phase1-auth-method rsa-signature
phase1-encrypt-algorithm 3des
phase1-hash-algorithm sha256
prf sha256
remote-address 38.38.38.38
traffic-selector { /Common/homer2 }
version { v1 v2 }
Impact:
Upgrading to version 15.1.0, which allows only one value for the version attribute, results in a failed upgrade/config load error.
Workaround:
Before upgrading, modify your config so that the version attribute has only one value for the version attribute.
Component: Access Policy Manager
Symptoms:
Issues with content displayed in iframes.
Conditions:
Using an iframe with the sandbox attribute in a web application.
Impact:
The web application does not work as expected.
Workaround:
Use an iRule as a workaround. Although the exact content of the iRule is strongly dependent on the web application, you can use the following iRule as a good example that can be customized:
when REWRITE_REQUEST_DONE {
HTTP::path] ends_with "/CUSOM_PATH_TO_JS_FILE"
REWRITE::post_process 1
when REWRITE_RESPONSE_DONE {
while {true} {
set str { sandbox="}
set strt [string first $str [REWRITE::payload]]
set str_len [string length $str]
$strt > 0
$strt < [expr [REWRITE::payload length] - $str_len]
REWRITE::payload replace $strt $str_len { sandbo1="}
} else {
break
Component: Access Policy Manager
Symptoms:
When the BIG-IP system is configured as a SAML IdP and processes an SAML Authentication Request, if it does not find the appropriate SAML SP connector, it does not log relevant information such as the Issuer, ACS _URL, and Protocol binding from the Authentication request.
Conditions:
This occurs when a BIG-IP system is configured as a SAML IdP and processes a SAML Authentication request, but does not find an appropriate SP configuration that matches the information provided in the SAML Authentication request.
Impact:
Troubleshooting the issue and fixing the SAML configuration is difficult since there is no relevant information in the error log.
Workaround:
Enable the log level for SSO to 'Debug', and capture the logs at the debug level to troubleshoot further.
Component: Application Security Manager
Symptoms:
Deception Response Pages is not inherited, but if common response pages are inherited, you are unable to save changes.
Conditions:
-- Deception Response Pages features licensed.
-- Parent policy selected with inheritance of response pages.
Impact:
Deception Response Pages cannot be modified.
Workaround:
None.
Component: Service Provider
Symptoms:
TMM might crash while attempting to connect internally to an internal virtual server (IVS) and the connection setup cannot be completed due to internal factors.
Conditions:
-- Extreme overload such that TMM is out of memory, or some other internal condition that prevents connection setup.
-- Connection to an internal virtual server is attempted.
Impact:
Traffic disrupted while tmm restarts.
Workaround:
None.
Component: Global Traffic Manager (DNS)
Symptoms:
The iRule 'drop' command may not drop a DNS response when called under DNS_RESPONSE event.
Conditions:
iRule drop is used under DNS_RESPONSE event.
Impact:
DNS response may be improperly forwarded to the client.
Workaround:
Use DNS::drop instead.
Component: Application Security Manager
Symptoms:
'Multiple of' validation of numeric values may not be correct in some scenarios.
Conditions:
-- Create default policy from API Security template.
-- Create default decimal parameter with 'Multiple of'=5.
-- Send request to /index.php?param=0.
Impact:
'Multiple of' validation of numeric values does not block as expected.
Workaround:
None.
Component: TMOS
Symptoms:
After upgrading, the configuration fails to load and throws an error about a profile that is located in profile_base.conf using SSLv2. However, upon inspection you will notice that there is no SSLv2 cipher in use.
Conditions:
The upgrade failure is seen when all the following conditions are met:
-- BIG-IP system with SSLv2 as the ciphers option in an SSL profile running software v12.x/v13.x.
-- Upgrading to a version that reports an error when using SSLv2, such as v14.x/v15.x.
(1) Modified root SSL profile (such as /Common/clientssl or /Common/serverssl) is present in bigip.conf.
(2) The modified root SSL profile contains an invalid keyword 'COMPAT', 'SSLv2', or 'RC2' in its ciphers
(3) The default profiles whose ciphers inherited from the root profile are not present in bigip.conf. The error for invalid ciphers is reported against these profiles.
Impact:
Beginning in version 14.x, SSLv2 has been changed from being a warning condition, and now prevents the configuration from loading. In most cases the upgrade script properly removes this, so there is no issue. However, if this issue is encountered, the configuration fails to load after upgrading.
Workaround:
There are two possible workarounds:
-- The easiest way to work around this is to comment out the modified base profile from bigip.conf and then run the command: tmsh load sys config.
-- If you are post upgrade, you can use sed to remove the !SSLv2 entries. To do so, perform these steps on the standby device:
1. cp /config/bigip.conf /config/backup_bigip.conf
2. Run: sed -i "s/\!SSLv2://g" /config/bigip.conf
3. tmsh load /sys config
Component: TMOS
Symptoms:
Malformed CSS where one of the style rules is missing a closing brace could cause LTM Rewrite profile to stop processing file or reset connection.
Conditions:
-- LTM Rewrite (URI translation) profile is attached to virtual server.
-- Content rewriting is enabled in Rewrite profile settings.
-- CSS file contains style rule with missing closing brace.
Impact:
URLs are not modified within affected files, starting from the missing closing brace. Intermittent connection resets occur.
Workaround:
Before rewriting, insert the missing symbol into CSS content either directly on the backend server or with an iRule.
Component: TMOS
Symptoms:
During EUDSF3.9.1 Front Port LED Test on BIG-IP iSeries platforms, the LCD buttons are unresponsive.
Conditions:
-- Running the EUDSF3.9.1 Front Port LED Test.
-- Using BIG-IP iSeries platforms.
Impact:
Cannot execute the EUDSF3.9.1 Front Port LED Test on BIG-IP iSeries platforms..
Workaround:
Continue to use EUDSF 3.7.0. EUDSF3.9.1 is required only for iSeries i10010, where the test does not have this issue.
Component: Local Traffic Manager
Symptoms:
BIG-IP may fail to update configuration of a virtual server when disabling or enabling spanning on the virtual address.
Conditions:
Spanning is disabled or enabled on a virtual address.
Impact:
Disabling or enabling spanning on a virtual address has no effect on the virtual-server configuration.
Depending on the configuration, virtual server may or may not forward the traffic when expected.
Workaround:
Enable/Disable spanning together with changing a traffic-group:
> modify ltm virtual-address 0.0.0.0 traffic-group none spanning enabled
> modify ltm virtual-address 0.0.0.0 traffic-group traffic-group-1 spanning enabled
Component: Local Traffic Manager
Symptoms:
MQTT monitor fails when non-default route domains are used.
Conditions:
-When a non-default route domain is configured for a pool member
-mqtt monitor in use
Impact:
Mqtt monitor does not work in route domain.
Component: TMOS
Symptoms:
In 2019, Brazil cancelled DST (Daylight Saving Time) and is now on standard time indefinitely. The BIG-IP system's time zone database needs to be updated to reflect this change.
Conditions:
None.
Impact:
BIG-IP systems configured to use "America/Sao_Paul" (or other applicable Brazilian localities) will still apply DST. Hence time will spring forward and backward on previously designated dates.
This will have no impact to application traffic handled by the BIG-IP system. However, logs, alerts, reports, cron jobs, etc. will use incorrect time.
Note: You can inspect the time changes your system is due to apply by running the following command from the BIG-IP system's advanced shell (bash):
zdump -v <timezone>
For example:
zdump -v America/Sao_Paulo
Workaround:
As a workaround, you can set the BIG-IP system's time zone to that of a different country with the same UTC offset and already not observing DST.
For example, instead of using "America/Sao_Paul", you could use "America/Buenos_Aires" to obtain the same result.
Component: Local Traffic Manager
Symptoms:
Due to a known issue, BIG-IP may create multiple connections for packets that should belong to a single flow. These connections will stay in the connection table until the idle timeout is reached. These connections can be used for forwarding the traffic.
Conditions:
BIG-IP may create multiple connections for packets that should belong to a single flow when both following conditions are true:
- Packets are coming at a very high rate from the network.
- Flow handling these packets is torn down.
Impact:
This might result in packets from the client being handled by one flow and packets from the server being handled by a different flow.
Component: TMOS
Symptoms:
SNMPv3 fails to read authenticated or encrypted messages to all but one of the members of a Config Sync group.
Conditions:
Using SNMPv3 to read or receive Traps from high availability (HA) pairs.
Impact:
SNMPv3 can only work for one member of a configsync group.
Configuring passwords on one device, makes that device work, but other members of the config sync group will now fail.
Workaround:
- check "Authoritative (security) engineID for SNMPv3" is not synced (mostly code released since 2019)
engineID needs to be unique per device
- Modify /defaults/config_base.conf to set sync to "no" and check that these do not sync
We must NOT sync these parameters as they need to match the individual device engineID
display-name "Authoritative (security) engineID for SNMPv3"
display-name "Authentication pass phrase for SNMPv3 messages"
display-name "Privacy pass phrase used for encrypted SNMPV3 messages"
display-name "User's passphrase"
display-name "Privacy passphrase"
### Mount usr as rw see see K11302
mount -o remount,rw /usr
pico /defaults/config_base.conf
# use Control-w to search for the display names above
# change "configsyncd yes" to "configsyncd no" if necessary in each location
# use Control-x y to exit with saving
# Restore usr as ro
mount -o remount,ro /usr
tmsh load sys config
Then once they are not syncing over, you can create v3 on each device using the same pass phrase as your SNMPv3 manager is using
tmsh modify sys snmp users add { v3snmp { auth-protocol sha privacy-protocol aes username mikev3 auth-password password3 privacy-password password3} }
tmsh modify sys snmp users modify { v3snmp { security-level auth-privacy access rw } }
Then each device should respond OK to query for that same pass phrase
snmpwalk -v 3 localhost -a sha -x aes -A password3 -X password3 -u mikev3 -l authpriv
For more information about SNMP, see the following articles.
K15681: Customizing the SNMP v3 engineID
K6821: SNMP v3 fails if the SNMP engine ID is not unique
K3727: Configuring custom SNMP traps
Component: Application Security Manager
Symptoms:
Deception Response pages shown and editable in parent policy.
Conditions:
-- Deception Response Pages feature is licensed and enabled.
-- You are editing the parent policy.
Impact:
Deception Response pages cannot be updated from the parent policy, thus any update of response pages fails with error.
Workaround:
None.
Component: Advanced Firewall Manager
Symptoms:
BIG-IP GUI users configured with the Application Security Administrator role are not allowed to manage DoS profile page and settings.
Conditions:
This affects users logged in with the Application Security Administrator role
Impact:
DoS profiles cannot be edited from the GUI.
Workaround:
You can use either workaround:
-- Change the user role to one that allows managing DoS profile.
-- Have the Application Security Administrator user edit profiles from tmsh.
Component: Device Management
Symptoms:
After upgrading BIG-IP software and reconfiguring high availability (HA), gossip may show 'UNPAIRED' and the REST endpoint /resolver/device-groups/tm-shared-all-big-ips/devices/ may show only one device.
Conditions:
-- This has been observed during upgrade from 14.x.x to 15.x.x.
-- Reconfigure HA.
-- View gossip results and REST endpoint output.
Impact:
The gossip output shows 'UNPAIRED', and the REST endpoint reports only one decide. SSL Orchestrator does not work as expected
Workaround:
If gossip shows 'UNPAIRED' after upgrade, you may need to do following at both devices:
1. Delete existing device information:
restcurl -X DELETE shared/resolver/device-groups/tm-shared-all-big-ips/devices
2. Force update:
bigstart restart restjavad
restcurl -X POST -d '{}' tm/shared/bigip-failover-state
Component: Local Traffic Manager
Symptoms:
Tmsh crash occurs rarely. The NGFIPS SDK may generate a core as well.
Conditions:
The exact conditions that trigger this are unknown.
It can be encountered when running the following tmsh command:
tmsh -a show sys crypto fips key field-fmt include-public-keys all-properties
Impact:
Tmsh may crash. You are exited from tmsh if you were using it as a shell.
Workaround:
None.
Component: Access Policy Manager
Symptoms:
Client browser traffic through APM Secure Web Gateway is being reset by APM SWG.
Conditions:
-- APM SWG transparent proxy is configured with captive portal. -- Client browser sends redirect to original URI on a TCP connection that was opened before access policy completion on captive portal.
Impact:
The connection is reset.
Component: Global Traffic Manager (DNS)
Symptoms:
Object to which the external monitor is attached is marked down.
Conditions:
Executing external monitors trying to access something without appropriate permissions.
Impact:
Object to which the external monitor is attached is marked down.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
Due to a known issue BIG-IP may advertise sub-optimal window size.
Conditions:
Result of (init-rwnd * client-mss) is greater than maximum window size (65,535).
Impact:
Degraded TCP performance.
Workaround:
Do not use init-rwnd values that might result in values higher than maximum window size (65,535).
Assuming MSS of 1480, the maximum value of init-rwnd is:
65535/1480 = 44.
Component: WebAccelerator
Symptoms:
HTTP 1.1 allows a conditional request with header If-Modified-Since or If-Unmodified-Since to determine whether a resource changed since a specified date and time. If AAM is provisioned and its policy is assigned to a virtual server, it may incorrectly respond with 304 Not Modified, even after the resource was updated.
Conditions:
-- AAM is provisioned and webacceleration policy is attached to a virtual server.
-- Client sends a conditional request with If-Modified-Since or If-Unmodified-Since header.
-- The BIG-IP system responds from AAM cache.
Impact:
Client does not receive an updated resource.
Workaround:
Use webacceleration profile without AAM policy for resources that require conditional checks falling back into Ramcache.
Component: Application Visibility and Reporting
Symptoms:
By building rpm of avrd, few cfg files get wrong set of permissions (executable)
Conditions:
Any build of avrd rpm
Impact:
Apparently not having the right set of permissions can lead to system halt
Workaround:
Change permissions on file:
# chmod -x /etc/avr/tmstat_tables.xml
Component: TMOS
Symptoms:
Features supported in newer versions of open-vm-tools will not be available.
Conditions:
This issue may be seen when running in VMware environments.
Impact:
Features that require a later version of open-vm-tools will not be available.
Workaround:
None.
Component: TMOS
Symptoms:
Prior to configuring an AWS WAF (AWAF) PAYG cloud instance, you may see errors related to an unlicensed LTM module. This occurs because the default provisioning for all instances is LTM nominal. However, the license associated with an AWAF PAYG cloud instance does not enable the LTM feature. As a result, the default provisioning for an unconfigured AWAF PAYG cloud instance will be incompatible with the PAYG license.
Conditions:
-- This issue relates only to AWAF PAYG cloud instances.
-- Not using the onboarding/templates to configure/provision the instance prior to use.
Impact:
Licensing error messages may be observed before the AWAF cloud instance is configured/provisioned. The functionality works as expected, however, so you can ignore these messages.
Workaround:
The recommended workflow for all cloud instances is to use onboarding/templates to configure/provision the instance prior to use. If this workflow is followed, any provisioning errors associated with the default provisioning are cleared prior to usage.
Component: Local Traffic Manager
Symptoms:
At validation time, mcpd issues a warning similar to the following:
warning mcpd[7175]: 01071859:4: Warning generated : /Common/test1:2: warning: ["\1" has no meaning. Did you mean "\\1" or "1"?][{(test) (\1)}]
Conditions:
Use arguments such as "\1", "\2", "\3" etc., in command regexp.
Impact:
A warning is generated, "\1" has no meaning, even though it is valid.
Workaround:
Ignore the warning.
Component: Local Traffic Manager
Symptoms:
Pool members are incorrectly marked DOWN by a monitor. The pool members send the expected response to the probe, but the BIG-IP system still marks them DOWN.
Conditions:
This issue occurs when all of the following conditions are met:
-- In-TMM monitoring is enabled on the system (the 'bigd.tmm' db key is set to 'enable'; note this is set to 'disable' by default).
-- One (or more) of your TCP or HTTP monitors specifies a receive string using HEX encoding, in order to match binary data in the server's response.
-- Depending on the HEX values specified (currently values in the range of 0x80-0xBF are believed to be affected), response matching fails.
Impact:
Objects that are meant to be marked UP are marked DOWN. As a result, no load balancing occurs to affected resources.
Workaround:
You can use either of the following workarounds:
-- Disable in-TMM monitoring by setting 'bigd.tmm' to 'disable'.
-- Do not monitor the application through a binary response (if the application allows it).
Component: TMOS
Symptoms:
On VCMP host, bcm56xxd crashes when it receives a corrupted DAG packets.
Conditions:
Unknown.
Impact:
Device goes offline, traffic disruption.
Component: Access Policy Manager
Symptoms:
TMM may crash.
Conditions:
-- APM session is established.
-- Network Access tunnel is established and used;
Impact:
APM end users experience Network Access tunnel disconnected. Traffic disrupted while tmm restarts.
Workaround:
None.
Component: Service Provider
Symptoms:
Diameter MRF persist records are not deleted at session timeout.
Conditions:
-- Diameter MRF in use
-- An unsuccessful CCR transaction occurs
Impact:
Unsuccessful transaction attempts do not time out and remain in the session database.
Workaround:
An iRule can be used to reset session timeout in diadb upon an unsuccessful transaction through MR_FAILED by executing DIAMETER::persist reset.
Component: TMOS
Symptoms:
Deployment of BIG-IP Virtual Edition may result in an error "Failed to generate ssh host key".
Conditions:
Azure only. Observed for instances with password-based authentication.
Impact:
A timing issues exists with host key generation. The Virtual Machine is likely to be deployed, but users and automation tools might be unable to communicate with the instance.
Workaround:
BIG-IP may still be accessible despite the error.
Component: TMOS
Symptoms:
If a very large amount of pool members are configured (tens of thousands), mcpd may miss its high availability (HA) heartbeat and be killed by sod.
Conditions:
-- A large number of pool members.
-- Pool member validation occurs (such as when loading a configuration or doing a configsync operation).
Impact:
Mcpd is killed by sod. This causes a failover (when the BIG-IP is in a DSC) or outage (if standalone).
Workaround:
None.
Component: TMOS
Symptoms:
-- Live install can fail at ~96% complete.
-- System memory might be exhausted, and the kernel terminates processes as a result.
Conditions:
-- During live-install.
-- Configuration roll-forward is enabled.
-- The uncompressed UCS size is larger than the available memory.
Impact:
The kernel terminates any number of processes; any/all critical applications might become nonfunctional.
Workaround:
You can use these two techniques to mitigate this situation:
-- Any file stored under /config is considered part of the configuration, so make sure there are no large, unnecessary files included in that directory.
-- If the configuration matches or is close to total system memory size, do not roll it forward as part of live install. Instead, save the UCS manually and restore it after rebooting to the new software.
To turn off config roll forward:
setdb liveinstall.saveconfig disable
For information about manually saving and restoring configurations, see K13132: Backing up and restoring BIG-IP configuration files with a UCS archive :: https://support.f5.com/csp/article/K13132.
Component: Application Security Manager
Symptoms:
Adding multiple user-defined Signatures via REST in quick succession may end with duplicate key database error.
Conditions:
At least two REST calls adding Attack Signatures and/or Attack Signature Sets which are sent in quick succession to the BIG-IP system.
Impact:
REST calls after the first may not be successful, resulting in failure to modify configuration as desired.
Workaround:
Retry the subsequent REST calls.
Component: Local Traffic Manager
Symptoms:
The BIG-IP system might close a connection when the HTTP response payload is modified and sent without chunking encoding. If communication goes over an HTTP connection, the BIG-IP system closes a connection to tell a client that a response is served. With HTTP/2 connections, an unsized response is marked with the END_STREAM flag. This case is not accounted for, and the BIG-IP system closes HTTP communication with a server anyway.
Conditions:
-- A virtual server has an http/2 profile configured on the client side.
-- There are other profiles configured which can modify the HTTP response payload.
Impact:
The BIG-IP system wastes resources not reusing a server side HTTP connection when an unsized response is sent over an HTTP/2 connection to a client.
Workaround:
None.
Component: TMOS
Symptoms:
Config load fails by issuing "tmsh load sys config verify"
01070734:3: Configuration error: Duplicate traffic selector is not allowed
Unexpected Error: Validating configuration process failed.
Conditions:
Duplicate IP addresses on multiple traffic-selectors attached to different ipsec-policies.
Impact:
Config load will fail after a reboot
Workaround:
Delete duplicate traffic-selectors.
Component: Access Policy Manager
Symptoms:
JWT token verification failure with error -1.
Conditions:
Token size larger than 4080.
Impact:
Request rejected and the log message simply says 'unknown error -1.'
Workaround:
None.
Component: TMOS
Symptoms:
TMM shows HSB transmit lockup message and cored.
Conditions:
-- Using an i4600 platform.
-- Other conditions under which this occurs are unknown.
Impact:
Disruption to processing traffic on the BIG-IP system.
Workaround:
None.
Component: TMOS
Symptoms:
In a VIPRION chassis, when the Primary blade is disabled (intentionally or due to an unexpected loss of functionality) and a new Primary blade is selected, there is a brief window of time during which status messages forwarded from MCPD on a Secondary blade to MCPD on the Primary blade might be dropped, possibly resulting in an incorrect view of the state of configured objects.
Conditions:
This problem may occur under the following conditions:
-- The state of a blade-local object/resources (such as a network interface or trunk) changes.
-- There is a high load on MCPD (for example, due to configuration reload on the new Primary blade) which delays processing of some MCPD actions.
Impact:
This problem may result in the state of blade-local objects (such as interfaces or trunks) being seen and reported incorrectly across the blades in the chassis, or on one or more specific blades (Primary, Secondary) in the chassis.
For example, if loss of the Primary blade results in one or more interfaces in a trunk being marked down by LACPD on a specific blade, resulting changes in trunk/member status may not be propagated correctly to the Primary blade, and from there to other Secondary blades.
Workaround:
None.
Component: TMOS
Symptoms:
Loading system configuration is failing after upgrade with an error message
01070619:3: Interface 5.0 media type is incompatible with other trunk members
Conditions:
-- Trunk interface created in BIG-IP version 12.3.4.
-- Trunk interfaces have different speeds (e.g. 100Mb interfaces and 1Gb interfaces)
-- Load the configuration after upgrading from v12.1.3.4 to v12.1.3.5.
Impact:
The system configuration is failing to load.
Workaround:
If you encounter this error, manually fix all trunks to only use interfaces of the same speed, and then load the configuration.
Component: TMOS
Symptoms:
The input validation performed by the BIG-IP system WebUI incorrectly allows the dns64-prefix option found in DNS profiles to include a subnet mask.
The WebUI should allow users to specify only a prefix (for example, 2001:db8:0:0:0:0:0:0 or 2001:db8::); however, it incorrectly allows users to specify a subnet mask too (for example, 2001:db8:0:0:0:0:0:0/96 or 2001:db8::/96).
In contrast, the TMSH utility correctly enforces values for this option.
Conditions:
The BIG-IP Administrator creates or modifies a DNS profile using the WebUI, and specifies an IP/SM value for the dns64-prefix option.
Impact:
Upon performing DNS64, TMM returns incorrect DNS answers that do not use the specified prefix. For example, if the Administrator specifies 2001:db8:0:0:0:0:0:0/96 as the prefix, and if the IPv4 address of the requested resource is 198.51.100.1, DNS64 returns ::198.51.100.1 instead of 2001:db8::c633:6401. This prevents end-user clients from reaching the intended resource.
The impact described in this section only applies to BIG-IP versions 14.1.0 and later. Previous BIG-IP versions also had this WebUI validation issue, but despite this TMM still returned the correct DNS answer.
Workaround:
When configuring this option using the WebUI, do not specify a subnet mask.
Component: Protocol Inspection
Symptoms:
If the IPS IM package is installed on a multi-slot device, and one slot is offline, the IM package version might be different on the offline slot when it comes back online.
It also shows different versions of the Active IM package on different slots.
Conditions:
-- Multi-bladed clustered system.
-- One of the blades is offline.
-- The IPS IM package is installed to the primary blade.
Impact:
The primary blade syncs the IM package to all of the secondary blades that are online; however, when the offline blade comes back online, it does not have the updated IM package.
As a result, traffic being processed by different blades will be using different IPS libraries and might cause inconsistency in the functionality
Workaround:
Although there is no workaround, you can prevent the issue by ensuring that all blades are online when you install an IPS IM package.
Component: Access Policy Manager
Symptoms:
JSON Web Token (JWT) verification failure with error -1.
Conditions:
Token size larger than 4080.
Impact:
Request is rejected.
Workaround:
None.
Component: Application Security Manager
Symptoms:
PRX.BRUTE_FORCE_* db tables do not have a row_limit, so they can grow to consume all available disk space in /var/lib/mysql.
Conditions:
ASM provisioned
Impact:
/var/lib/mysql can run out of disk space
Workaround:
1. Truncate the two large tables. This clears all the row in those table and should make disk space.
Note that existing brute force username and IPs reporting data will be lost.
# mysql -u root -p$(perl -MPassCrypt -nle 'print PassCrypt::decrypt_password($_)' /var/db/mysqlpw) -e "TRUNCATE TABLE PRX.BRUTE_FORCE_MITIGATED_USERNAMES"
# mysql -u root -p$(perl -MPassCrypt -nle 'print PassCrypt::decrypt_password($_)' /var/db/mysqlpw) -e "TRUNCATE TABLE PRX.BRUTE_FORCE_MITIGATED_IPS"
2. Add row_limit for the two tables to avoid the same issue in the future.
Add following lines in the bottom of this file, /etc/ts/tools/clean_db.yaml
PRX.BRUTE_FORCE_MITIGATED_USERNAMES:
row_limit: 100000
order_by: brute_force_mitigated_username_id
PRX.BRUTE_FORCE_MITIGATED_IPS:
row_limit: 100000
order_by: brute_force_mitigated_ip_id
Restart clean_db process (there is no impact of restarting this process)
# pkill -f clean_db
Wait 30 sec, and make sure the process came back
# ps aux | grep clean_db
Component: Local Traffic Manager
Symptoms:
When SSL::enable is issued in an iRule, for example in the HTTP Request event, it will not enable the server side profile if the server side profile is disabled.
Conditions:
An HTTP profile is configured on a virtual, and the server-ssl profile on the same virtual is disabled.
Impact:
The connection will close instead of completing.
Workaround:
Do not use a disabled server-ssl profile in this situation.
Component: TMOS
Symptoms:
During initialization of very large configurations it is possible that the watchdog timer will fire and reset the bcm56xxd driver.
Conditions:
System configuration with very large number of objects being loaded.
Impact:
The driver restarts.
Component: Local Traffic Manager
Symptoms:
TMM crashes with a SIGFPE and restarts. The TMM logs contain the following panic message:
Assertion "xbuf_delete_until successful" failed.
Conditions:
This issue occurs when the following conditions are met:
-- The system has been configured with a standard virtual server and an Any IP (ipother) virtual server chained together. This can be done explicitly using an iRule that features the 'virtual' command to connect the two virtual servers, or implicitly with certain APM configurations.
-- The pool member on the server-side asks this specific virtual server configuration on the BIG-IP system to retransmit data.
Impact:
Traffic disrupted while tmm restarts.
Workaround:
Ensure the target virtual server in the chain configuration does not use the ipother profile.
Component: Local Traffic Manager
Symptoms:
The HTTP/1.1 request/response statistic fields in the HTTP profile are incremented incorrectly when HTTP2 traffic is encountered.
There is not currently a way to view the HTTP2 and HTTP3 request/response stats on the HTTP profile.
Conditions:
-- Client or server sends HTTP2 request/response.
-- Using GUI, TMSH, iControl (SOAP), or SNMP.
Impact:
Incorrect HTTP/1.1 request/response statistic values are present in the HTTP profile when HTTP2 traffic is encountered.
Workaround:
None.
Component: TMOS
Symptoms:
When you modify a virtual server, the GUI returns an error similar to the following:
01b90011:3: Virtual Server /Common/vs2_udp's Traffic Matching Criteria /Common/vs2_udp_VS_TMC_OBJ illegally shares destination address, source address, service port, and ip-protocol with Virtual Server /Common/vs1_tcp destination address, source address, service port.
Conditions:
This occurs when either of the following occur:
-- When renaming the virtual server.
-- When changing the address-list attribute.
Impact:
Cannot update virtual configuration with new value.
Workaround:
None.
Component: Application Security Manager
Symptoms:
Incorrect parameter parsing occurs under some conditions. For example, in a signature violation, the 'Actual Parameter Name' value appears as 'attachment; filename'.
Conditions:
-- ASM in use.
-- Request contains multipart headers.
Impact:
Incorrect policy enforcement.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
If the VLAN that services the state mirror connection between BIG-IP systems is renamed, it can cause a disruption of the state mirror connection. It can also lead to an eventual crash.
Conditions:
Renaming the VLAN that services the state mirror connection between BIG-IP systems in an high availability (HA) configuration.
Impact:
System might crash eventually.
Workaround:
Do not rename the VLAN that services the state mirror connection between BIG-IP systems in an HA configuration.
Component: Global Traffic Manager (DNS)
Symptoms:
Selected prober pool not visible on server or data center in Internet Explorer or Edge
Conditions:
This occurs when you have a prober pool configured for a data center or server, and you are viewing them in the GUI using Internet Explorer or Edge.
Impact:
The prober pool is not displayed.
Workaround:
Use Chrome or Firefox as browser
Component: TMOS
Symptoms:
After starting up, mcpd may have defunct child "run" and "xargs" processes
Conditions:
Slow disk storage or large configuration files.
Impact:
Minimal; some zombie processes are created.
Component: Local Traffic Manager
Symptoms:
Pool members are marked down from the in-TMM monitor.
Conditions:
Device has a large number of in-TMM monitors.
Impact:
Monitor target may appear down when it is actually up.
Workaround:
Disable in-tmm monitors:
tmsh modify sys db bigd.tmm value disable
Component: Global Traffic Manager (DNS)
Symptoms:
DNSSEC does not insert NSEC3 records for NXDOMAIN responses.
Conditions:
-- "process-xfr yes" is set for the dns profile associated with the listener;
-- There is no "Zone Transfer Clients" nameserver configured for that zone.
-- There is no wideip configured.
Impact:
DNSSEC does not respond NSEC3 for non-existent domain.
Workaround:
1. Change this setting for dns profile from "process-xfr yes" to "process-xfr no";
2. Add a nameserver for "Zone Transfer Clients" of that zone.
3. Add a wideip.
Component: TMOS
Symptoms:
BIG-IP systems might not respond to ( ARP / Neighbour Discovery ) requests received via EtherIP tunnels on a multi-blade system.
Conditions:
Decapsulated ( ARP / Neighbour Discovery ) requests for an address owned by the BIG-IP system is processed by a secondary blade.
Impact:
Some endpoints may not be able to resolve ( ARP / Neighbour protocol ) via EtherIP tunnel.
Workaround:
Create static ARP entries on affected endpoints.
Component: TMOS
Symptoms:
LTM high availability (HA) sync group are syncing GTM zone configuration changes.
Conditions:
1. BIG-IPs has both LTM and GTM provisioned.
2. The two BIG-IPs are inside one LTM sync group.
Impact:
GTM zone files are accidentally modified.
Component: TMOS
Symptoms:
Cannot scp copy a file to the BIG-IP system. The system reports an error:
path not allowed
Conditions:
Issue occurs when both conditions are present:
-- The BIG-IP user has 'shell tmsh' or 'shell none' access.
-- The scp destination is the real path target (not listed in the 'allow' list) of a symbolic link that is listed in the scp 'allow' list (/config/ssh/scp.whitelist).
For example:
scp to /var/tmp succeeds.
scp to /shared/tmp fails with 'path not allowed'.
Impact:
Cannot copy files to a path present under whitelist.
Workaround:
Use the explicitly listed (symlink) path as the scp destination.
Component: TMOS
Symptoms:
Part of the HSB becomes unresponsive, and there is no logging of additional registers to assist in diagnosing the failure.
Conditions:
It is unknown under what conditions the HSB becomes unresponsive.
Impact:
Limited visibility into the HSB state when it becomes unresponsive.
Workaround:
None.
Component: Access Policy Manager
Symptoms:
When running ldbutil utility, if the '--list' option is specified, then the '--instance' option has no effect. All the local users will be listed.
Conditions:
When both '--list' and '--instance' options are specified.
Impact:
The output lists all the local users and not limiting to the '--instance' option given.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
Issue is reported at the following system setup:
client <-> BIG-IP <-> concentrator <-> proxy <-> BIG-IP nat gateway <-> Internet
-- SYN Cookie got activated on F5 nat gateway.
-- Latency from 'Internet' (public host) is observed at 'Proxy' device sitting before F5 nat gw.
-- During the latency issue, SYN Cookie was active and evicting connections.
-- When SYN Cookie is enabled, it switches to l7 delayed binding as expected but it is not sending ACK for HTTP request so the client sends it again and again.
Conditions:
Haredware SYN Cookie is enabled on FastL4 profile
Impact:
High latency is observed.
Workaround:
Disable the SYN Cookie on the FastL4 profile
Component: Local Traffic Manager
Symptoms:
With in-tmm monitoring enabled (or sys db bigd tmm set to enable) and with https monitor's ssl-profile set to none, the expected behavior is to send ciphers in ClientHello based on default serverssl profile as mentioned in GUI help for https monitor.
Conditions:
Configure HTTPS Monitor with ssl-profile "None".
Impact:
Ciphers are not exchanged as expected in the ClientHello Packets
Workaround:
Configure HTTPS Monitor without ssl-profile option, default serverssl profile will be used
Component: TMOS
Symptoms:
You see the following errors in /var/log/ltm:
err mcpd[8985]: 0107071c:3: Trouble allocating mac address for vlan object /Common/external.
Conditions:
Conditions under which this occurs are unknown.
Impact:
There is no known impact to the system as a result of this log message.
Workaround:
If this reoccurs, you can try force reloading mcpd.
For more information, see K13030: Forcing the mcpd process to reload the BIG-IP configuration, available at https://support.f5.com/csp/article/K13030.
Component: TMOS
Symptoms:
In the GUI, the Virtual Server screen displays the available address-lists or port lists for source address, but there is no clarity on whether the options are selected or available.
Conditions:
-- Virtual server's source address section.
Impact:
If you do not make a selection and try to create the Virtual Server, an error occurs: An error has occurred while trying to process your request.
Workaround:
Click to select the address-list of port-list displayed as source address for Virtual Server.
Component: Local Traffic Manager
Symptoms:
-- The virtual-address (and virtual server) will no longer work.
-- The BIG-IP won't answer ARP requests for it.
-- Loading the config again or performing similar operations will not re-create the virtual-address.
Conditions:
-- A virtual address is deleted while it is in use by an address list and virtual server.
-- MCPD is restarted (or the unit rebooted, etc.).
Impact:
Traffic processing is disrupted
Component: Advanced Firewall Manager
Symptoms:
The AFM Auto threshold and behavioral dos historical data synchronization process consumes greater than 90% CPU. This affects TMM performance and some outages may occur.
Conditions:
This occurs in both high availability (HA) and standalone configurations. In both cases "MCPD" issues were reported. (Delay in response or the daemon crashed)
Impact:
TMM performance is affected and outages may occur.
Workaround:
Kill the AFM data synchronization process:
kill -9 afm_cmi.py
Component: TMOS
Symptoms:
Using an iControl PUT command to modify a virtual address will change that address's netmask to ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff.
Conditions:
Modifying a virtual address using an iControl PUT command.
Impact:
An unintentional change to the virtual address's netmask.
Workaround:
Two options:
-- Use a PATCH command instead of a PUT command.
-- Always specify the netmask explicitly when making changes.
Component: TMOS
Symptoms:
OVSDB-server fails to make SSL connections when Selinux is enforced.
In /var/log/openvswitch/ovsdb-server.log:
...|00012|stream_ssl|ERR|/config/filestore/files_d/Common_d/certificate_d/:Common:myCert_2468_1: stat failed (Permission denied).
Conditions:
-- Navigate to System :: Configuration : OVSDB.
-- Add cert and keys.
Impact:
Permission denied, SSL connection failure.
Workaround:
Step 1: Check openvswitch SELinux denial:
# audit2allow -w -a
Example output:
type=AVC msg=audit(1566915298.607:32958): avc: denied { search } for pid=18966 comm="ovsdb-server" name="/" dev="dm-7" ino=2 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:f5config_t:s0 tclass=dir
Was caused by:
Missing type enforcement (TE) allow rule.
&nbnbsp; You can use audit2allow to generate a loadable module to allow this access.
Step 2: Find openvswitch components that need Linux policy additions:
# audit2allow -a
Example output:
#============= openvswitch_t ==============
allow openvswitch_t f5config_t:dir search;
allow openvswitch_t f5filestore_t:dir search;
allow openvswitch_t f5filestore_t:file { getattr open read };
Step 3: Modify the policy to allow access to the component openvswitch_t:
# audit2allow -a -M openvswitch_t
Step 4: Apply the policy:
# semodule -i openvswitch_t.pp
Component: Local Traffic Manager
Symptoms:
After upgrading SSL Orchestrator to version 5.x, traffic gets reset, SSL Orchestrator hangs, and tcpdump analysis indicates that connections are being reset due to SSL handshake timeout exceeded.
Conditions:
-- SSL Orchestrator configured.
-- Initial plaintext traffic resembles SSLv2 hello message or has less-than-enough bytes for SSL to process.
Impact:
SSL Orchestrator hangs on that connection, unable to bypass traffic until the connection times out. Other connections handle traffic during this interval.
Workaround:
None.
Component: iApp Technology
Symptoms:
If lengthy Tcl errors are displayed in the GUI while creating an iApp, the GUI can hang.
Conditions:
-- Creating an iApp that contains a syntax error.
-- A large error message is emitted.
Impact:
GUI hangs.
Workaround:
Restart the tomcat process:
tmsh restart sys service tomcat
Component: TMOS
Symptoms:
You are unable to change the management IP from the GUI
Conditions:
This is encountered when using the GUI to change the management IP address via the System :: Platform page.
Impact:
The GUI indicates that it will redirect you to the new IP address. You will eventually be redirected but the management IP address is not changed on the BIG-IP device.
Workaround:
Use tmsh to create the management IP. This will overwrite the old one.
Example:
create /sys management-ip [ip address/prefixlen]
To view the management IP configurations
tmsh list /sys management-ip
Component: Service Provider
Symptoms:
When a georedundancy profile is attached to a TCP, UDP, and SCTP virtual server, proxy does not convert to GEO proxy.
Conditions:
Attach georedundancy profile to a standard TCP, UDP, or SCTP virtual server.
Impact:
Unable to convert proxy to GEO proxy when georedundancy profile is attached with TCP, UDP and SCTP virtual
Workaround:
None.
Component: Access Policy Manager
Symptoms:
SSO setting for Portal Access (PA) resource assigned to any webtop section cannot be updated due to the following error:
No such atomic attribute:name in class:webtop_section_webtop_section_resource
Configuration with such a resource cannot be transferred to another BIG-IP system due to the same error.
Conditions:
BIG-IP system configuration with the following objects:
-- SSO configuration (any type).
-- PA resource with resource item.
-- The webtop section with the PA resource.
-- Full webtop.
-- Per-session access policy with resource assignment agent which assigns webtop, webtop section, and PA resource.
Impact:
Configuration cannot be updated or transferred.
Workaround:
Delete webtop sections from the configuration and re-create them after transfer / upgrade / update process.
Component: TMOS
Symptoms:
Connections that are hardware accelerated and that use asymmetric routing may use the wrong MAC address for return traffic. This can be observed by looking at a packet capture.
Conditions:
Hardware acceleration is enabled (ePVA/fastL4) with asymmetric routing.
Impact:
The return traffic has the wrong source MAC address. This may affect packet forwarding depending on the configuration.
Workaround:
Disable HW acceleration.
Component: Local Traffic Manager
Symptoms:
RST_STREAM is sent on a serverside stream in closing state in HTTP/2 full proxy.
Conditions:
-- HTTP/2 clientside and serverside profiles are attached to the virtual server.
-- HTTP and httprouter profiles are attached to the virtual server.
-- Race between clientside and serverside closing, where clientside closes faster.
Impact:
RST_STREAM is sent on a stream in closed state.
Component: TMOS
Symptoms:
During re-licensing or license reload, an unknown trap OID 1.3.6.1.2.1.47.2.0.1.0.1 may be sent.
Conditions:
Occurs during license reload or reactivation.
Impact:
After a license reload, the unknown trap can be seen like the following:
run "tcpdump -ni mgmt port 162 -vvvv &":
12:01:59.883331 IP (tos 0x0, ttl 64, id 47411, offset 0, flags [DF], proto UDP (17), length 101)
10.248.136.179.55540 > 172.28.8.68.snmptrap: [bad udp cksum 0x486e -> 0xd7b8!] { SNMPv2c { V2Trap(58) R=1205683810 .1.3.6.1.2.1.1.3.0=1775555 .1.3.6.1.6.3.1.1.4.1.0=.1.3.6.1.2.1.47.2.0.1.0.1 } }
Component: Local Traffic Manager
Symptoms:
Child FastL4 profile is being reset after clicking Update from GUI.
Conditions:
-- Create child SSL FastL4, profile inheriting settings from a parent FastL4 profile.
-- From the command line, change any of the CLI-only visible settings in the child FastL4 profile (e.g., pva-acceleration, explicit-flow-migration, etc.), and save the changes.
-- In the GUI, click the Update button in the child FastL4 profile without making any change.
Impact:
The operation overwrites the CLI changes made in the child profile, and inherits those values from the parent settings instead.
Workaround:
None.
Component: TMOS
Symptoms:
When a node (Disabled by Monitor) is updated via the member screen (no change to configuration required), the status changes from:
'Available (Disabled) pool members is available, monitor disabled'.
'Available (Disabled), pool member is available, user disabled'.
Conditions:
-- A node disabled by Monitor.
-- Go to GUI LTM pool member and navigate into the monitor disabled member, then update without any configuration change.
Impact:
Pool member goes to 'user-disabled'.
Workaround:
To recover, re-enable the pool member.
Component: TMOS
Symptoms:
Multicast route entries are not populating in tmm after failover. ZebOS has the multicast entries, but tmm does not.
Conditions:
-- High Availability (HA) configured, with multicast traffic.
-- A failover occurs.
Impact:
Multicast traffic does not pass through properly
Workaround:
Clear the multicast entries in ZebOS manually:
> clear ip mroute *
> clear ip igmp group
Component: Advanced Firewall Manager
Symptoms:
Logs/Tmctl shows packet dropped whereas AVR shows Action as 'Allowed' and not 'Dropped'.
Conditions:
-- AFM configured.
-- AFM passes the message to AVR for reporting.
Impact:
The operation does not update the drop flag. It appears from AVR Reporting that packets are allowed, but actually they are dropped
Workaround:
There is no workaround at this time.
Component: Global Traffic Manager (DNS)
Symptoms:
The virtual server for an LTM redundant peer is continually updated with its IP/Port changing back and forth between two values, leading to perpetual GTM configuration syncs.
Conditions:
The destination IP:port of the virtual server on the LTM is not in sync between the LTM devices in the device-group.
Impact:
The virtual server is flapping status between "blue" and 'green', and its destination IP:port is changing between a correct value and an incorrect one. Traffic will be impacted.
Workaround:
Perform a configsync on the LTM device-group that owns the virtual server.
Component: Carrier-Grade NAT
Symptoms:
IPv4 Packets are forwarded to server-side with destination address changed to LTM pool member address even when 'translate-address disabled' is configured on a NAT64 virtual server.
Conditions:
-- Create iRules for LTM pool selection.
-- Configure the NAT64 virtual server with 'translate-address disabled'.
-- Send IPv6 client request accessing the NAT64 virtual server.
Impact:
Server-side IPv4 packets are forwarded with destination address modified. The server-side packets do not reach the intended destination, resulting in connection failures.
Workaround:
Use normal LTM pool selection instead of iRules-based, LTM pool selection.
Component: TMOS
Symptoms:
When there are more than 10000 lines in /var/log/secure* files, visiting System :: Logins :: [History|Summary] in the GUI shows 'No Entries' instead of the actual error message about the large number of lines.
Conditions:
If there are more than 10000 lines in /var/log/secure* files.
Impact:
GUI displays 'No Entries' instead of the actual error message.
Workaround:
-- Via the CLI by specifying the number of lines:
tmsh show sys log security lines 15000 | less
-- Delete the large amount of secure files from /var/log/.
Component: Local Traffic Manager
Symptoms:
When the client attempts to connect, even when sending the proper SNI extension, the BIG-IP system resets the connection after the client hello.
Conditions:
-- SNI Required set to true.
-- No Server Name configured in the client SSL profile.
Impact:
SSL connections with valid SNI are closed, and the client cannot connect. With generic alerts enabled, you will see 'SSL alert number 40'. This is because the system does not read the server names from the SAN extension within the certificate.
Workaround:
Specify a valid server name in the server name field of the client SSL profile.
Component: TMOS
Symptoms:
When mcpd starts up, unmonitored pools in an high availability (HA) group do not contribute to the HA group's score.
Conditions:
-- HA group configured with at least one pool.
-- At least one of the pools assigned to the HA group is not using monitoring.
-- mcpd is starting up (due to bigstart restart, or a reboot, etc.).
Impact:
Incorrect HA Group score.
Workaround:
Remove the unmonitored pools from the HA group and re-add them.
Component: TMOS
Symptoms:
Log message that indicates this issue may happen:
session_reply_multi: ERROR: unable to send session reply: ERR_BOUNDS
[...] valid s_entry->ref_cnt
Conditions:
-- Specific MRF configuration where all 500 session entries are owned by a single tmm.
-- High rate of session lookups with a lot of entries returned.
Note: This issue does not affect HTTP/2 MRF configurations.
Impact:
TMM core: the program terminates with signal SIGSEGV, Segmentation fault. Traffic disrupted while tmm restarts.
Workaround:
1. Change MRF configuration to spread session lookups across multiple tmms.
2. Reduce the sub-key entries to far below 500.
Component: Application Visibility and Reporting
Symptoms:
AVRD can crash repeatedly when configured to send telemetry data externally.
Conditions:
-- AVR is configured to send telemetry data to an external source (like connection with BIG-IQ).
-- Large number of config objects in the system, such as virtual servers and pool members.
Impact:
AVRD process crashes, and telemetry data is not collected.
Workaround:
Split the configuration updates into smaller batches
Component: TMOS
Symptoms:
Hertfordshire county is missing from Regions in the United Kingdom Country/State list.
Conditions:
-- Creating a GTM region record.
-- Attempting to select Hertfordshire county for the United Kingdom.
Impact:
Cannot select Hertfordshire county from United Kingdom Country/State list.
Workaround:
None.
Component: TMOS
Symptoms:
Under low-disk conditions for the /var/ filesystem, BIG-IP system processes may incorrectly update root's crontab file (/var/spool/cron/root). This results in the file contents being removed; i.e., the file is empty.
Conditions:
Low disk space on the /var filesystem.
Impact:
System and user entries in root's crontab file stop executing.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
When using a variable as the only parameter to the iRule persist command, for example:
when HTTP_REQUEST {
set persistence none
persist $persistence
The iRule validation fails with the message:
Persistence mode (Cookie) called out in rule <rule name> requires a corresponding persistence profile for virtual server
Conditions:
Using a variable as the only parameter to the iRule persist command.
Impact:
Validation fails and hence the system config cannot be loaded.
Workaround:
The first parameter is one of pre-defined action keywords, so use plain text.
Component: TMOS
Symptoms:
MCP DB is not updated after loading a UCS file.
Conditions:
1. Save UCS with 'flow-control' default value 'tx-rx'.
2. Modify the value from 'rx-tx' to 'none'.
3. Save another UCS with modified value.
4. Load the UCS with default value, everything works fine here.
5. Load the UCS with the modified value.
Impact:
The 'flow-control' setting gets changed. The functionality does not work after the first UCS load as MCP DB is not getting updated.
Workaround:
Load the same UCS again.
The MCP DB gets updated properly.
Component: TMOS
Symptoms:
Upgrade failure when loading configuration file or ucs from older version, with the below error message against child client-ssl profile:
Client SSL profile (/Common/child): must have at least one set of CA type cert-key-chain.
Conditions:
The issue occurs when all of the following conditions are met:
-- When upgrading from an older version (earlier than 14.0.0) to a newer version (14.1.0 or later).
-- The configuration has a child client SSL profile that inherits from a parent client SSL profile.
-- The parent SSL profile has SSL forward proxy enabled and proxy-ca-cert/proxy-ca-key configured.
Impact:
Unable to upgrade the system with old configuration.
Workaround:
You can workaround this issue using the following procedure:
1. Manually edit /config/bigip.conf to add the following lines:
proxy-ca-cert /Common/rsa.crt
proxy-ca-key /Common/rsa.key
2. Reload the configuration using the following command:
tmsh load sys config
Here is an example:
ltm profile client-ssl /Common/child {
app-service none
cert /Common/default.crt
cert-key-chain {
default {
cert /Common/default.crt
key /Common/default.key
chain none
defaults-from /Common/parent
inherit-certkeychain true
key /Common/default.key
passphrase none
proxy-ca-cert /Common/rsa.crt <===== add this line
proxy-ca-key /Common/rsa.key <===== add this line
Component: Application Security Manager
Symptoms:
When Bot Defense is used on the backend server that overrides request cookies, requests to non-HTML resources may fail, or may receive the whitepage JavaScript challenge. An example is when a back-end server responds with a Set-Cookie header containing empty values for each cookie request cookie it does not recognize.
Conditions:
-- Bot Defense is enabled.
-- Backend server is overriding the Bot Defense cookies with the TS prefix.
Impact:
Some URLs fail to load following the JavaScript challenge.
Workaround:
Use an iRule to strip the TSPD_101 cookie from the request before forwarding it to the backend:
when HTTP_REQUEST_RELEASE {
HTTP::cookie remove "TSPD_101"
Component: Local Traffic Manager
Symptoms:
'tmsh load sys config verify' reports error messages for a signed iRule:
-- 01071485:3: iRule (/Common/example) content does not match the signature.
-- Unexpected Error: Validating configuration process failed.
Conditions:
-- Have an iRule which ends in a comment.
-- iRule is signed.
-- Run the command:
tmsh load sys config verify
Impact:
'tmsh load sys config verify' reports an error when there should be none. Configuration validation fails.
Workaround:
Delete or move any comments that are at the end of the iRule, and validate the configuration again.
Component: TMOS
Symptoms:
After running 'promptstatusd -y' to check current failover status, it displays an incorrect Active/Standby status in the CLI prompt.
Conditions:
This occurs under the following conditions:
1. Modify the db variable: bigdb failover.state.
2. Check that /var/prompt/ps1 and CLI prompt reflect the setting.
2. Reboot the BIG-IP system.
Impact:
Status shown in the prompt does not change.
Workaround:
Do not run 'promptstatusd -y' command manually.
The db variable 'failover.state is a status-reporting variable. The system does not report status manually set to something other than the actual status.
Note: 'promptstatusd' is not a BIG-IP user command, it is a daemon. It is highly unlikely that manually running this command will produce information that is useful or relevant to the status being sought.
Component: TMOS
Symptoms:
When loading the configuration, if the virtual address is disabled but the virtual server is enabled, the virtual server may still pass traffic.
Conditions:
-- Loading the configuration.
-- A virtual server's virtual address is disabled.
Impact:
Virtual servers unexpectedly process traffic.
Workaround:
Manually re-enable and disable the virtual address.
Component: TMOS
Symptoms:
The LTM MySQL health monitor fails to connect to a MySQL server running MySQL Server v8.0.
A pool member configured for a MySQL server running MySQL Server v8.0 and using the MySQL health monitor will be marked DOWN.
Conditions:
This occurs when using the LTM MySQL health monitor to monitor a MySQL server running MySQL Server v8.0.
Impact:
BIG-IP cannot monitor the health of a MySQL server running MySQL Server v8.0 using the MySQL health monitor.
Component: TMOS
Symptoms:
The 'sys httpd ssl-certkeyfile' tmsh help text contains a reference to bigpipe, which is no longer supported.
Conditions:
Viewing tmsh help for 'sys httpd ssl-certkeyfile'.
Impact:
Incorrect reference to bigpipe.
Workaround:
You can use the following command sequence to change the key:
modify httpd { ssl-certfile [string] ssl-certkeyfile [string] }
Component: Local Traffic Manager
Symptoms:
If a FastL4 virtual server receives an ICMP response without first receiving an ICMP request, the checksum on the ICMP response that is egressed by tmm will not be calculated correctly.
Conditions:
An ICMP response without a corresponding ICMP request, such as in non-symmetric routing scenarios.
Impact:
ICMP replies are forwarded with the incorrect checksum and likely will be dropped by the recipient or other devices on the network.
Workaround:
Ensure symmetric routing. Configure L7 virtual servers for use with ipother profiles.
Component: TMOS
Symptoms:
An upgrade or UCS restore fails on the host with an error message:
err mcpd[1001]: 01071769:3: Decryption of the field (sym_unit_key) for object (<guest name>) failed.
Conditions:
-- An upgrade or UCS restore of the vCMP host.
-- Having a vCMP guest's sym-unit-key field populated.
-- Having changed the host's master key.
Impact:
The upgrade or UCS restore fails with an MCPD error.
Workaround:
Comment out the sym-unit-key field and load the configuration.
Component: Local Traffic Manager
Symptoms:
For a database (mssql, mysql, postgresql or oracle) monitor type, with a 'recv' string configured, a pool member configured to use the DB monitor may be marked down even if the server is working and includes the configured response string among the response data.
Debug logging of the SQL monitor indicates the following:
... [DBPinger-3778] - Response from server: Database: 'db1'Database: 'information_schema'
... [DBPinger-3778] - Checking for recv string: information_schema
... [DBPinger-3778] - Analyze Response failure
The log shows 'Analyze Response failure' error message even when the configured 'recv' string appears within the response message from the DB server.
Conditions:
This occurs when the string matching the configured 'recv' string value does not appear in the response from the DB server in the row indicated by the 'recv-row' value configured for the monitor.
The default value of 'none' for the 'recv-row' monitor configuration value is actually interpreted as 'row 1' by the DB monitor core implementation.
Therefore, with the default configuration, any 'recv' string configured must appear in the first row of the DB server response in order to be recognized as a match.
Impact:
The DB monitor fails, and the DB server (node) is marked as down even though it is reachable and responding correctly per the configured 'recv' string.
Workaround:
You may use one of the following methods to work around this issue:
1. Configure the DB monitor's 'recv' string to match on the first row in the server response message.
2. Configure the 'recv-row' value in the DB monitor to match the row of the DB server's response which contains the configured 'recv' string.
3. Do not configure 'send' or 'recv' string for the DB monitor.
Component: TMOS
Symptoms:
While doing an snmpwalk, you encounter the following error:
-- tcl callback Default return string: istats: tmstat_open_read: open: /var/tmstat/istats: Permission denied.
-- istats: tmstat_read: open: /var/tmstat/istats: Permission denied.
-- ERROR opening iStats read segment '/var/tmstat/istats': Permission denied.
Conditions:
This occurs when using SNMP to access iStats.
Impact:
iStats cannot be accessed through SNMP and generates an error.
Workaround:
None.
Component: TMOS
Symptoms:
An incorrect MSS value might be used when hardware (HW) syncookies are used, and the MTU is smaller than the MSS.
Conditions:
-- The BIG-IP system sends TCP segments, fragmented across multiple IP packets, that exceed the size of the local interface MTU.
-- This occurs when HW Syncookies are enabled.
Impact:
TCP segments larger than the local interface MTU sent towards the client. These TCP segments are transmitted as IP fragments.
Workaround:
Increase MTU size.
Component: TMOS
Symptoms:
In reboot case, the BIG-IP system buffers the shutdown sequence log messages and writes them to disk once the syslog service starts during the boot process. The boot_marker message is written before shutdown messages sync to disk. This leads to out-of-sequence log messages, making it difficult to determine when the service stop occurred.
Conditions:
Reboot the BIG-IP system.
Impact:
Log messages appear out of order. It is difficult to tell whether service stop happened as part of reboot, or any error during the subsequent boot process.
Workaround:
None.
Component: Advanced Firewall Manager
Symptoms:
Flow Inspector cannot filter on IP address with non-default route_domain.
Conditions:
-- In Flow Inspector.
-- Attempting to filter results.
-- Some results use IP addresses with non-default route domains.
Impact:
Filter does not return results as expected.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
Source-port preserve-strict and OneConnect may result in zombie forwarding flows.
Conditions:
-- Source-port is set to preserve-strict.
-- OneConnect configured.
Impact:
Zombie forwarding flows. Over time, the the current allocation count grows and does not return to its prior level when traffic stops.
Workaround:
None.
Component: Application Security Manager
Symptoms:
Manual changes to a policy imported as XML may introduce corruption for Response Pages. The following log appears:
ASM subsystem error (asm_config_server.pl ,F5::PrepareConf::Policy::prepare_alternate_response_file_tbl): failed to parse response headers - please check response page.
Conditions:
-- XML policy file is missing a response header.
-- Import the policy.
Impact:
The affected reponse page is not returned for traffic as expected, and an error is reported instead.
Workaround:
Mitigation:
Ensure that response_header exists in XML policy file before import.
Workaround:
Go to the affected policy's Response Pages: Login Page, click Save and then click Apply Policy.
Component: Local Traffic Manager
Symptoms:
Virtual server iRule does not match an External Data Group key that is 128 characters long.
Conditions:
-- A string type External Data Group with a key/value pair whose key is 128 characters long.
-- An iRule using [class match] to get the value from the Data Group.
Impact:
The call to [class match] returns an empty string ("").
Workaround:
None.
Component: TMOS
Symptoms:
If Sflow agent loses the connection with the SNMPD Manager, it tries to connect multiple times but fails to reconnect.
Conditions:
Sflow agent loses connection with the SNMPD Manager. The conditions that may trigger this are unknown.
Impact:
Snmpd service restarts repeatedly
Workaround:
Run 'tmsh restart sys service sflow_agent' to clear the session data in the sflow agent which results in successful re-connection with snmpd.
Component: TMOS
Symptoms:
It is possible with large configurations to make SNMP requests that require a lot of processing to gather the statistics needed to respond to the request.
Conditions:
With large configurations, it is possible to overburden MCPD and SNMPD such that client queries time out.
Impact:
SNMP clients might think the BIG-IP system has become unresponsive.
Workaround:
If the responses to SNMP queries are taking too long, MCPD and SNMPD may overburden the control plane. It may be necessary to lengthen the timeout and retry values used by the SNMP client. It may also be helpful to trim what is queried, for example, not repetitively walking large tables like the Virtual Server or LTM Pool Member tables for statistics.
Component: TMOS
Symptoms:
vCMP host or guest is upgraded, and the vCMP guest is 'Inoperative', with messages similar to the following in /var/log/ltm:
-- warning clusterd[1546]: 013a0005:4: Clusterd using /VERSION for SW specification.
-- info clusterd[1546]: 013a0023:6: Blade 1: No info received from slot: Starting up
-- err clusterd[1546]: 013a0004:3: result {
-- err clusterd[1546]: 013a0004:3: result.code 17237812
-- err clusterd[1546]: 013a0004:3: result.attribute float_mgmt2_ip
-- err clusterd[1546]: 013a0004:3: result.message 01070734:3: Configuration error: Cluster alt-address: 192.168.1.246 cannot be the same address family as cluster address: 192.168.1.246
-- err clusterd[1546]: 013a0004:3: }
-- err clusterd[1546]: 013a0004:3: Per-invocation log rate exceeded; throttling.
-- notice clusterd[1546]: 013a0006:5: Disconnecting from mcpd.
-- info clusterd[1546]: 013a0007:6: clusterd stopping...
Conditions:
-- Isolated vCMP guest.
-- Both 'Address' and 'Alt-Address' are assigned the same IPv4 address.
-- Upgrade occurs.
Impact:
Upon host/guest upgrade, vCMP guest is 'Inoperative'.
Workaround:
There are 3 workarounds:
-- Assign IPv4 management-ip for the guest PRIOR to upgrade:
tmsh modify vcmp <guest_name> management-ip <ip>/<mask>
-- Prior to upgrade, assign IPv6 'Alt-Address' within the guest:
tmsh modify sys cluster default alt-address <IP>/<mask>
-- When already upgraded and seeing the issue on a guest: execute:
> /shared/db/cluster.conf
The config loads, and /shared/db/cluster.conf is recreated:
cat /shared/db/cluster.conf
cluster default {
alt_addr="192.168.1.246/24" ====>
min_up_members="1"
min_up_members_enable="disable"
min_up_members_action="failover"
addr 192.168.1.246/24 ====>
members {
software HD1.1 "BIG-IP 14.1.0.3 0.0.6 0.0.6" true
software HD1.2 " " false
software HD1.3 " " false
Component: Local Traffic Manager
Symptoms:
Under certain conditions, a pool member's cur_sessions stat may increase, but not decrease when it should.
Conditions:
- The virtual server using the pool has an iRule attached that references global variables.
- The virtual server using the pool has an ASM security policy attached to it.
- Traffic flows to the pool member.
Impact:
Incorrect stats.
Component: TMOS
Symptoms:
An unrelated iControl REST transaction error message is returned when committing an iControl REST transaction that does not contain any tasks:
Error: Missing transaction ID for this call.
Conditions:
-- Committing an iControl REST transaction.
-- The task does not contain any tasks within 120 seconds of creating the transaction.
Impact:
Unrelated error message can be confusing and increase troubleshooting time.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
Filtering IPv4 routes using route-map does not work. All the IPv4 redistributed routes fail to redistribute if the route-map is attached to the OSPFv3 IPv4 address-family.
Conditions:
1. Configure two OSPF sessions, one for the IPv4 address-family and the other for the IPv6 address family.
2. Redistribute kernel routes.
3. Check routes are propagated.
4. Add a route map to allow any IPv4 kernel route matching IP address.
Impact:
All routes fail to propagate and show that the IPv6 OSPF database external is empty. All IPv4 routes are blocked to redistribute instead of the routes mentioned in the route-map/prefix-list.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
On a single virtual server, when 'TLS Renegotiation' is enabled in an associated Client SSL profile, the system should prevent enabling the 'Enforce TLS Requirements' option in the associated HTTP/2 profile.
Conditions:
BIG-IP system validation does not prevent this configuration in the following scenario:
1. Disable the 'Enforce TLS Requirements' option in the HTTP/2 profile.
2. Enable 'TLS Renegotiation' in the Client SSL profile.
3. Enable the 'Enforce TLS Requirements' option in the HTTP/2 profile.
Impact:
The configuration will not load if saved.
Workaround:
Do not simultaneously disable 'Enforce TLS Requirements' in the HTTP/2 profile, and enable 'TLS Renegotiation' in the Client SSL profile on a single virtual server.
Component: Local Traffic Manager
Symptoms:
A CMP (Clustered Multiprocessing) state change occurs when the state of the BIG-IP system changes.
This happens in the following instances:
- Blade reset.
- Booting up or shutting down.
- Running 'bigstart restart'.
- Setting a blade state from/to primary/secondary.
During these events, there is a small chance that ingress ACK packet of previously established BGP connection is going to be disaggregated to the new processing group(TMMs) and selected TMM is ready to process traffic, but is not ready yet to process traffic for existing connection. In this case, connection isn't processed and reset instead.
Conditions:
-- VIPRION chassis with more than one blade.
-- CMP hash of affected VLAN is changed from the Default value, for example, to Source Address.
-- BGP peering is configured.
-- CMP state change is occurred on one of the blades.
-- BGP ingress ACK packet is disaggregated to TMM, which either wrong TMM or not ready to process the packet of already established connection
Impact:
Affected BGP peering is reset and dynamic routes learnt by the configured protocol are withdrawn, making it impossible to advertise dynamic routes of affected routing protocols from the BIG-IP system to the configured peers. This can lead to unexpected routing decisions on the BIG-IP system or other devices in the routing mesh.
In most cases, unexpected routing decisions are from networks learnt by affected routing protocols when the routing process on the BIG-IP system becomes unreachable. However, this state is short-lived, because the peering is recreated shortly after the routing protocol restarts. The peering time depends on the routing configuration and responsiveness of other routing devices connected to the BIG-IP system. It's the usual routing convergence period, which includes setting the peering and exchanging routing information and routes.
Workaround:
There is no workaround, but the issue was never seen with a configuration where CMP hash of affected VLAN is changed back to Default value.
Component: Local Traffic Manager
Symptoms:
On a high availability (HA) paired device group configuration, where there are FQDN nodes as pool members in a pool, when the pool member is enabled or disabled on one device, and with config-sync, the other device does not fully update the peer. The template node gets updated with the new value, but the ephemeral pool member retains the old value.
Conditions:
Steps to Reproduce:
1. Configure HA, specifically a Device group (e.g., Failover) with two BIG-IP systems.
2. Create an HTTP pool (TEST_FQDN_POOL) and FQDN Pool Member on both systems.
3. Wait for the FQDN pool member to report as AVAIL_GREEN and the ephemeral node as AVAIL_BLUE on both systems.
4. Tmsh login to any of the systems.
5. Run the command:
tmsh run cm config-sync to-group Failover
6. Run the command:
tmsh modify ltm pool TEST_FQDN_POOL members modify { example.com:http { session user-disabled } }
7. Run the command:
tmsh run cm config-sync force-full-load-push to-group Failover
Impact:
FQDN pool member enabling/disabling is not being fully propagated to the other device after config-sync.
Workaround:
None.
Component: Application Security Manager
Symptoms:
TMM might crash and generate a core file when using a Bot Defense profile that is configured for multiple domains.
Conditions:
Bot Defense is configured with multiple 'Related Site Domains' and attached to a virtual server.
Impact:
TMM crash with core. Traffic disrupted while tmm restarts.
Workaround:
None,
Component: TMOS
Symptoms:
When a user with the admin role attempts to save a UCS with a passphrase, the following error is encountered:
[resource-admin@inetgtm1dev:Active:Standalone] ucs # tmsh save sys ucs /var/local/ucs/test-ucs passphrase password
Saving active configuration...
Error: Bad encryption password. <=========
Operation aborted.
/var/tmp/configsync.spec: Error creating package
WARNING:There are error(s) during saving.
Not everything was saved.
Be very careful when using this saved file!
Error creating package
Error during config save.
Unexpected Error: UCS saving process failed.
Conditions:
1) Log into the BIG-IP system as a user with admin role that has Advanced Shell access.
2) Attempt to create a UCS with a passphrase.
Impact:
Unable to save UCS with a passphrase.
Workaround:
This affects users logged in with the Admin role; you will be able to create a UCS with a passphrase while logged in as either the root user or as a user with the resource-admin role.
Component: TMOS
Symptoms:
While loading the /Common partition, config routing protocols on other partition route-domains will be removed.
Conditions:
-- Configure route-domains on other partitions with routing-protocols.
-- Load the /Common partition config alone.
Impact:
Routing protocols config from other partitions are removed.
Workaround:
Reload the config with the command:
load sys config partitions all
Component: Global Traffic Manager (DNS)
Symptoms:
The following tmsh command allows you to delete all wide IPs using an 'all' specifier:
modify gtm distributed-app da1 wideips delete { all }
There is no equivalent iControl REST operation to do this.
Conditions:
This can be encountered while trying to delete all wide IPs from a distributed application via iControl REST.
Impact:
iControl REST calls that should allow you to remove all wide IPs from a GTM distribution application return an error, leaving you unable to complete the task via iControl REST.
Workaround:
You can use one of the following workarounds:
-- Use the WebUI.
-- Use the tmsh utility, for example:
tmsh modify gtm distributed-app da1 wideips delete { all }
-- Invoke tmsh from within the bash iControl REST endpoint, for exmaple:
curl -u username:password -s -H 'Content-Type: application/json' -X POST -d "{\"command\":\"run\",\"utilCmdArgs\":\"-c 'tmsh modify gtm distributed-app da1 wideips delete { all }'\"}" https://<IP>/mgmt/tm/util/bash
Component: TMOS
Symptoms:
If a VLAN, VLAN group, or tunnel has a name with more than 15 characters, BGP does not function properly on that interface.
Conditions:
-- BGP Dynamic routing in use.
-- Interface name greater than 15 characters.
Impact:
BGP Dynamic Routing is not working.
Workaround:
1. Rename the interface using 15 or fewer characters.
2. Remove Static Binding and Bind to all interfaces.
Component: Access Policy Manager
Symptoms:
Email sent from APM is not readable in some languages.
Conditions:
APM administrator has configured Email Agent in the per-session policy.
Impact:
Users receiving the email in certain languages cannot read the email.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
The bigd.mgmtroutecheck db variable can be enabled to prevent monitor traffic from going through the management interface (for information, see K14549: Overview of the 'bigd.mgmtroutecheck' database key :: https://support.f5.com/csp/article/K14549); however, if in-tmm monitors are configured, the setting will be ignored after a bigstart restart.
Conditions:
-- bigd.mgmtroutecheck is enabled
-- bigd.tmm is enabled (i.e., in-tmm monitors are configured).
-- tmm has a route configured to the management interface.
-- A pool member exists that matches a route through the management interface.
-- bigstart restart is performed.
Impact:
In-tmm monitor traffic uses the management interface if there is a route to the pool member via the management interface, even when bigd.mgmtroutecheck indicates it is enabled.
Workaround:
Component: Local Traffic Manager
Symptoms:
Memory may occasionally leak when a software crypto request is cancelled before it has completed.
Conditions:
There are a number of reasons why a software crypto request may be canceled.
Impact:
Memory may leak.
Workaround:
No workaround.
Component: TMOS
Symptoms:
Network Map in GUI shows incorrect [Forced Offline] status.
Conditions:
-- Multi-blade system
The device status in the network map is falsely shown [Forced Offline] when actual device status is something else other than [Forced Offline]. In other words, it is always shown as [Forced Offline].
-- Non multi-blade system
The device status in the network map is falsely shown [Forced Offline] when actual device status is something else other than [Active] or [Forced Offline]. In other words, it displays fine only for [Active] and [Forced Offline].
Impact:
The device status in the network map is not reliable
Workaround:
None.
Component: Application Visibility and Reporting
Symptoms:
One thread of the avrd process spontaneously starts to consume 100% CPU.
Conditions:
The exact conditions under which this occurs are unknown, but might occur only on vCMP configurations.
Impact:
System performance degrades.
Workaround:
Restart TMM:
bigstart restart tmm
Component: TMOS
Symptoms:
With certain modules provisioned, the following debug apmd log events are reported to the logging server:
-- slot1/systemname debug apmd[14204]: GSSAPI client step 2.
-- err apmd[14204]: 01490107:3: ...: Local error, SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Ticket expired) (-2)
Conditions:
-- APM, APML, ASM, AVR, iRulesLX, LTM provisioned.
-- Configured to use Generic Security Service Application Program Interface (GSSAPI, also GSS-API).
Impact:
Debug-level messages are logged even when the BIG-IP system is not configured to use debug logging:
debug apmd[14204]: GSSAPI client step 2.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
VLAN groups do not bridge non-link-local multicast traffic.
Conditions:
-- VLAN groups configured.
-- Using non-link-local multicast traffic.
Impact:
Non-link-local multicast traffic does not get forwarded.
Workaround:
None.
Component: Application Security Manager
Symptoms:
When an authentication request with Authorization: Negotiate arrives to ASM. ASM does not count it as a login attempt. As a result brute force protection isn't applied.
Conditions:
-- ASM provisioned.
-- ASM policy attached to a virtual sever.
-- Login URL configured in ASM policy.
-- Brute force protection enabled in ASM policy.
Impact:
Brute force attack checking can be skipped if the backend server authorization type is NTLM but the client sends 'Authorization: Negotiate TlR'.
Workaround:
Use iRule which changes 'Authorization: Negotiate TlR' to NTLM on the client side (before ASM) and sets is back to the original value on the server side (after ASM)
Component: Local Traffic Manager
Symptoms:
If the BIG-IP system is configured in L2Wire mode, packets from srcIP 0.0.0.0 are dropped.
Conditions:
L2Wire mode.
Impact:
All srcIP 0.0.0.0 packets are dropped silently.
Workaround:
Configure the virtual server to be in L2-forward mode.
Component: TMOS
Symptoms:
Secondary active blades go offline.
Conditions:
-- Cluster with three or more active blades.
-- Primary blade is rebooted.
For example, on a 4-bladed system, after slot 1 (primary blade) was rebooted and slot 2 (secondary blade) takes over as primary, slots 3 and 4 both go offline due to high availability (HA) table, with the logs showing reason as 'waiting for configuration load'.
Impact:
Cluster reduced to a single blade, which may impact performance.
Workaround:
None.
Component: Global Traffic Manager (DNS)
Symptoms:
Under certain circumstances, DNSSEC zone SEPS records may be wiped out from running configuration.
Conditions:
This occurs only with GTM configurations loaded by the command: tmsh load sys config gtm-only.
Impact:
SEPS records may be lost after a configuration reload.
Workaround:
None.
Component: TMOS
Symptoms:
The validation of SNMPv3 user names is not strict, and allows users of both the GUI and TMSH to enter badly formed user names. When the SNMP daemon reads these user names from the snmpd.conf file, validation rejects the names.
Conditions:
Poorly formed SNMPv3 user names can be entered into configuration, for example, names with embedded spaces.
Impact:
The user names are not accepted by the SNMP daemon when it reads the configuration from the snmpd.conf file.
Workaround:
Use alphanumeric characters for SNMPv3 user names, and do not include embedded spaces in the names.
Component: Local Traffic Manager
Symptoms:
If the cluster management IP address is not in the list of failover network unicast addresses, the blade management IP addresses in the unicast mesh will not be able to receive failover messages from peer devices.
Conditions:
-- VIPRION chassis or vCMP guest on a VIPRION chassis.
-- Per-blade management IP addresses listed in the failover network unicast mesh.
-- No cluster management IP address listed.
Impact:
The blade management IP addresses in the failover network unicast mesh stop functioning:
[root@VIP2200-R75-S5:/S1-green-P::Standby:In Sync] config # tmctl -w 200 -S sod_tg_conn_stat
entry_key local_failover_addr remote_device_name pkts_received transitions last_msg status
----------------------------- ------------------- ------------------------------ ------------- ----------- ---------- ------
10.200.75.8->10.10.10.1:1026 10.10.10.1:1026 VIP2200-R75-S8.sin.pslab.local 3249 3 1555399271 1
10.200.75.8->10.200.75.3:1026 10.200.75.3:1026 VIP2200-R75-S8.sin.pslab.local 0 1 0 0 <--
10.200.75.8->10.200.75.4:1026 10.200.75.4:1026 VIP2200-R75-S8.sin.pslab.local 0 1 0 0 <--
Workaround:
You can add an explicit management IP firewall rule to allow this traffic:
tmsh modify security firewall management-ip-rules rules add { accept_udp_1026 { place-before first ip-protocol udp destination { ports add { 1026 } } action accept } }
This will add a firewall policy so port 1026 is no longer locked down, and the blade management IP addresses in the unicast mesh should begin to function properly.
Component: TMOS
Symptoms:
TMSH allows you to create multiple access records with the same IP protocol, same Source IP network, and same community string.
Conditions:
Duplicate access records are created in TMSH.
Impact:
Unintended permissions can be provided when an undesired access record with the correct community string is matched to a request instead of the desired access record.
Workaround:
Use the Configuration Utility to manage SNMP v1/2c access records. (The GUI properly flags the error with the message:
The specified SNMP community already exists in the database.
If you use tmsh, ensure that community strings remain unique within each Source IP Network for each IP protocol.
Component: TMOS
Symptoms:
It is possible, although unlikely, for a vCMP host to scan the /shared/vmdisks directory for virtual disk files while the directory is unmounted.
As such, virtual disk files that existed before the reboot will not be detected, and the vCMP host will proceed to create them again.
The virtual disks get created again, delaying the guests from booting. Once the guests finally boot, they have no configuration.
Additionally, the new virtual disk files are created on the wrong disk device, as /shared/vmdisks is still unmounted.
Symptoms for this issue include:
-- Running the 'mount' command on affected host blades and noticing that /shared/vmdisks is not mounted.
-- Running the 'tmsh show vcmp guest' command on affected host blades (early on after the reboot) and noticing some guests have status 'installing-vdisk'.
-- Running the 'lsof' command on affected and unaffected host blades shows different device numbers for the filesystem hosting the virtual disks, as shown in the following example (note 253,16 and 253,1):
qemu-kvm 19386 qemu 15u REG 253,16 161061273600 8622659 /shared/vmdisks/s1g2.img
qemu-kvm 38655 qemu 15u REG 253,1 161061273600 2678798 /shared/vmdisks/s2g1.img
-- The /var/log/ltm file includes entries similar to the following example, indicating new virtual disks are being created for one of more vCMP guests:
info vcmpd[x]: 01510007:6: VDisk (s2g1.img/2): Adding.
info vcmpd[x]: 01510007:6: VDisk (s2g1.img/2): Syncing with MCP - [filename:s2g1.img slot:2 installed_os:0 state:0]
notice vcmpd[x]: 01510006:5: Guest (s2g1): Creating VDisk (/shared/vmdisks/s2g1.img)
info vcmpd[x]: 01510007:6: VDisk (s2g1.img/2): Syncing with MCP - [filename:s2g1.img slot:2 installed_os:0 state:1]
info vcmpd[x]: 01510007:6: Guest (s2g1): VS_ACQUIRING_VDISK->VS_WAITING_INSTALL
info vcmpd[x]: 01510007:6: Guest (s2g1): VS_WAITING_INSTALL->VS_INSTALLING_VDISK
notice vcmpd[x]: 01510006:5: Guest (s2g1): Installing image (/shared/images/BIGIP-12.1.2.0.0.249.iso) to VDisk (/shared/vmdisks/s2g1.img).
info vcmpd[x]: 01510007:6: VDisk (s2g1.img/2): Syncing with MCP - [filename:s2g1.img slot:2 installed_os:0 state:2]
Conditions:
-- VIPRION chassis provisioned in vCMP mode with more than one blade in it.
-- Large configuration with many guests.
-- The VIPRION chassis is rebooted.
-- A different issue, of type 'Configuration from primary failed validation' occurs during startup on one or more Secondary blades. By design, MCPD restarts once on affected Secondary blades, which is the trigger for this issue. An example of such a trigger issue is Bug ID 563905: Upon rebooting a multi-blade VIPRION or vCMP guest, MCPD can restart once on Secondary blades.
Impact:
-- Loss of entire configuration on previously working vCMP guests.
-- The /shared/vmdisks directory, in its unmounted state, may not have sufficient disk space to accommodate all the virtual disks for the vCMP guests designated to run on that blade. As such, some guests may fail to start.
-- If you continue using the affected guests by re-deploying configuration to them, further configuration loss may occur after a new chassis reboot during which this issue does not happen. This occurs because the guests would then be using the original virtual disk files; however, their configuration may have changed since then, and so some recently created objects may be missing.
Workaround:
There is no workaround to prevent this issue. However, you can minimize the risk of hitting this issue by ensuring you are running a software version (on the host system) where all known 'Configuration from primary failed validation' issues have been resolved.
If you believe you are currently affected by this issue, please contact F5 Networks Technical Support for assistance in recovering the original virtual disk files.
Component: Local Traffic Manager
Symptoms:
Configuring a network virtual server with a source address list causes the system to treat the virtual server as a host.
Conditions:
-- Configure a source address list on the virtual server.
-- Configure a network address for the destination of the virtual server (not an address list).
Impact:
Traffic does not flow to the virtual server as expected.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
Traffic is not passing in virtual wire when virtual server type is configured as standard, protocol set to "All Protocols" and the IP profile is ipother.
Conditions:
-- Virtual wire is configured
-- Virtual server type is standard
-- IP profile is ipother
Impact:
Virtual wire traffic matching the virtual server is dropped.
Component: TMOS
Symptoms:
APM may intermittently drop log messages, leading to missing information on policy execution or other events.
Conditions:
Using APM per-request policies, or ACCESS::log iRule commands.
Impact:
Administrator may fail to report certain logging events, hindering troubleshooting or auditing efforts.
Workaround:
No workaround is possible. When reviewing APM logs, keep in mind that during periods of high activity (greater than 100 log messages in 1-to-2 seconds) that the system may drop some log messages.
Component: Local Traffic Manager
Symptoms:
When the server connection fails to be established due to server being down or actively rejecting the connection, LB_FAILED should fire and allow a new destination to be selected via iRule.
Conditions:
- iRule with LB_FAILED event
- server connection establishment fails
Impact:
Selection of a new destination via LB_FAILED is not possible, thus the client connection will be aborted.
Workaround:
No workaround available.
Component: Protocol Inspection
Symptoms:
Automatic hitless upgrade for protocol inspection fails on vCMP guests. This occurs because vCMP guest don't install f5_api_com key and certificates.
Conditions:
After licensing a vCMP guest, there is no f5_api_com key or certificate (you can run key_cache_path and crt_cache_path to determine that).
Impact:
Hitless upgrade fails for protocol inspection and traffic classification on vCMP guests.
Workaround:
Install the hitless upgrade IM package manually.
Component: TMOS
Symptoms:
When tcpdump is invoked with the epva option on a non-epva platform (BIG-IP Virtual Edition, for example), it fails with an unclear message
errbuf:DPT Provider fatal error. Provider:ePVA Provider. No valid arguments.
Conditions:
-- Using a non-epva platform such as VE.
-- Calling the epva option:
+ Directly:
tcpdump -i 0.0 --f5 epva
+ Indirectly using 'all' (which includes epva):
tcpdump -i 0.0 --f5 all
Impact:
Unclear message does not give clear indication what the issue is, or how to get tcpdump to run with the 'all' option on non-epva platforms
Workaround:
Do not use the explicit epva option on non-epva platforms (it does not work anyway, as there is no epva debug information on those platforms).
Instead of 'all', explicitly specify other, non-epva providers on such platforms, for example, specifying 'noise' and 'ssl' providers:
tcpdump -i 0.0 --f5 n,ssl
Component: TMOS
Symptoms:
MCP expects a monitoring response from SQL server and starts polling for data continuously, resulting in infinite loop.
Conditions:
In one of the corner cases of SQL monitoring, MCP expects to read monitoring data from the PostgreSQL server, but there is no data available to read.
Impact:
MCPD goes into an infinite loop and skips the heartbeat report, resulting in its restart. While MCPD is restarting, the system is offline and does not process traffic. After restart, system operation returns to normal.
Workaround:
None.
Component: TMOS
Symptoms:
'Login failed' is displayed on the BIG-IP system's login screen.
Conditions:
-- Remote authentication is enabled.
-- There are more than three name servers configured.
Impact:
Admins may not be able to log into the BIG-IP GUI with their admin user account if the first 3 configured DNS name servers are not reachable.
Workaround:
None.
Component: TMOS
Symptoms:
Restjavad frequently (approximately every 5 minutes) restarting due to OutOfMemory:Java heap space with no extra memory.
Conditions:
-- BIG-IP system with no extra memory given to restjavad.
-- The configuration contains a large number of configuration items related to APM access-policies, APM policy-items, APM policy agents, LTM nodes, LTM rules, DNS Requests, sys application services, LTM data-groups, LTM profiles, security bot-defense profiles, and sys file ssl-certs.
Impact:
REST API intermittently unavailable.
Workaround:
Give restjavad extra memory. This is two-step process.
1. Update memory allocated to the control plane using TMUI. System :: Resource Provisioning. The line for Management has a drop-down box for Small, Medium, or Large. The resulting sizes for restjavad is 192, 352, and 592, respectively. Set this to Large unless a custom value greater than 500MB is already in use.
2. Run the following two commands, in sequence:
tmsh modify sys db restjavad.useextramb value true
bigstart restart restjavad
Component: TMOS
Symptoms:
After restarting httpd using the iControl REST API, httpd fails to start, even with a subsequent restart of httpd at the command line.
Similar to the following example:
config # restcurl -u admin:admin /tm/sys/service -X POST -d '{"name":"httpd", "command":"restart"}'
"kind": "tm:sys:service:restartstate",
"name": "httpd",
"command": "restart",
"commandResult": "Stopping httpd: [ OK ]\r\nStarting httpd: [FAILED]\r\n(98)Address already in use: AH00072: make_sock: could not bind to address n.n.n.n:n\nno listening sockets available, shutting down\nAH00015: Unable to open logs\n"
config # tmsh restart sys service httpd
Stopping httpd: [ OK ]
Starting httpd: [FAILED]
Conditions:
Restarting httpd service using iControl REST API.
Impact:
Httpd fails to start.
Workaround:
To recover from the failed httpd state, you can kill all instances of the httpd daemon and start httpd:
killall -9 httpd
tmsh start sys service httpd
Component: TMOS
Symptoms:
A user account which may have originally been manually configured as a local user (auth user) but may have since been removed, might still get authenticated and be able to modify the BIG-IP configuration.
Conditions:
-- User account configured as local user.
-- The user account is deleted later.
(Note: The exact steps to produce this issue are not yet known).
Impact:
The deleted user that no longer exists in the local user list and which is also not explicitly authorized by remote role groups, can get authenticated. The deleted user is also able to modify the BIG-IP configuration via iControl.
Workaround:
None.
Component: Global Traffic Manager (DNS)
Symptoms:
mcpd is in a restart loop after creating an internal DNSSEC FIPS key on a secondary GTM while rebooting the primary DNSSEC key generator GTM (gtm.peerinfolocalid==0).
Conditions:
New DNSSEC internal FIPS key is created and assigned to DNSSEC zone when BIG-IP system with gtm.peerinfolocalid==0 is down.
Impact:
mcpd is in a restart loop.
Workaround:
For maintenance window operations, set DNSSEC peer leader to the unit that will remain UP while rebooting the primary key generator in sync group (gtm.peerinfolocalid==0).
# tmsh modify gtm global-settings general peer-leader <gtm-server-name>
After the reboot is complete, all devices are back up, and everything looks good in the configs, clear the peer-leader setting:
# tmsh modify gtm global-settings general peer-leader none
If there are two GTM units: GTM1 (having gtm.peerinfolocalid == 0), GTM2, and you are going to reboot GTM1, then before rebooting, run the following command to configure the DNSSEC peer-leader setting:
# tmsh modify gtm global-settings general peer-leader GTM2
After reboot, clear the peer-leader setting:
# tmsh modify gtm global-settings general peer-leader none
Component: TMOS
Symptoms:
On an SNMPv3 configuration, when a security-name and a username are the same but have different passwords, traps are not properly crafted.
Conditions:
security-name is the same as an SNMPv3 username.
Impact:
SNMP traps cannot be decoded
Workaround:
Delete or rename user.
Component: Local Traffic Manager
Symptoms:
The BIG-IP system sends multicast ping from a disabled blade. tmm core
Conditions:
-- There is one or more blades disabled on the VIPRION platform.
-- VLAN failsafe is enabled on one or more VLANs.
-- the VLAN failsafe-action is set to 'failover'.
-- There is more than one blade installed in the chassis or vCMP guest.
Impact:
The BIG-IP system sends unexpected multicast ping requests from a disabled blade.
Workaround:
To mitigate this issue, restart tmm on the disabled blade. This causes tmm to stop sending the multicast traffic.
Impact of workaround: Traffic disrupted while tmm restarts.
Component: TMOS
Symptoms:
The GUI, is not displaying LTM policies created with a rule in which log criteria is 'action when the traffic is matched'.
Also, some of the Actions disappear while adding multiple actions in a rule.
Using tmsh shows the polices created in the GUI.
Conditions:
From the GUI, create LTM policies with a rule in which log criteria is 'action when the traffic is matched'.
Impact:
GUI is not displaying LTM policies created with log as action in rule.
Workaround:
Use tmsh.
Component: iApp Technology
Symptoms:
While creating a UCS file, you see a confusing error message, and the UCS file is not created:
Failed task: %s: %s"%(taskUri, taskResult['message']))"
Conditions:
This can be encountered while trying to create a UCS file.
Impact:
Certain failure messages are not interpreted correctly by the script, resulting in the actual error message not being displayed.
Workaround:
None.
Component: TMOS
Symptoms:
When two BIG-IP systems are paired, it is possible during startup for the network connection to be made too early during the boot sequence. This may leave a peer in the inoperative state.
Conditions:
This is a timing-related issue that might occur during boot up of high availability (HA) peers.
Impact:
An HA peer does not become ACTIVE when it should.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
Repeated TMM service crash SIGBUS with memory copy operation at the top of stack trace.
Conditions:
TMM loads filestore file and size of this file is smaller than the size reported by mcp or if this ifile store is not present at all.
This condition is possible due to
- filesystem errors/corruption or
- BIG-IP user intervention.
Filesystem error might be due to power loss, full disk or other reasons.
Impact:
TMM crash.
The program terminated with signal SIGBUS, Bus error.
Workaround:
Manual copy of the "good" ifile store and forceload on the previously bad unit. Usually trivial, but error prone.
Another workaround is clean install, if possible/acceptable
Component: TMOS
Symptoms:
Upon querying a sysTmmStat* SNMP OID (for example, sysTmmStatTmUsageRatio5s), you find your SNMP client returns an error message similar to the following example:
No Such Instance currently exists at this OID
The very next time you query that same SNMP OID (or any other sysTmmStat* SNMP OID), you find they all work as expected and return the correct result.
Conditions:
This issue occurs after restarting only the mcpd daemon, i.e., running bigstart restart mcpd.
Impact:
All sysTmmStat* SNMP OIDs do not work until one of them is queried at least once, and the query is allowed to fail. After that, all sysTmmStat* SNMP OIDs work as expected.
Workaround:
Restart all services together, i.e., running the command: bigstart restart.
Should the mcpd daemon happen to be restarted on its own, you can simply ignore the error message and allow your SNMP polling station to fail a single polling cycle.
If you want to ensure that this issue does not occur, for example, so that your SNMP polling station does not generate unnecessary alarms, do not restart the mcpd daemon on its own, but rather restart all services together by running the following command:
bigstart restart
Component: Local Traffic Manager
Symptoms:
If an iRule is being referenced by another iRule, and the reference is then removed, attempts to delete the formerly referenced iRule will result in an error similar to the following:
01070265:3: The rule (/Common/irule1) cannot be deleted because it is in use by a rule (/Common/irule2).
Conditions:
-- An iRule referencing another iRule.
-- The referencing iRule is in use.
Impact:
Unable to delete the iRule.
Workaround:
Save and re-load the configuration.
Component: Local Traffic Manager
Symptoms:
RESOLVE::lookup returns empty string.
Conditions:
Input bytes array is at length of 4, 16, or 20.
For example:
[RESOLV::lookup @dnsserveraddress -a [binary format a* $host1.d1test.com]]
Impact:
RESOLVE::lookup returns empty string.
Workaround:
Use lindex 0 to get the first element of the array.
For example:
[RESOLV::lookup @dnsserveraddress -a [lindex [binary format a* $host1.d1test.com] 0]]
Component: Application Security Manager
Symptoms:
ASM blocks a legal request and fires CSRF false positive violations when csrf JavaScript code is injected into a page without an html tag.
Conditions:
-- ASM provisioned.
-- ASM policy attached to a virtual server.
-- CSRF protection configured.
-- HTML pages learning features enabled.(BruteForce/WebScraping).
-- CSRF JavaScript code is injected into a page without an html tag.
Impact:
HTTP requests are blocked sometimes when they should not be.
Workaround:
To workaround this issue, configure asm internal and then restart asm, as follows:
/usr/share/ts/bin/add_del_internal add cs_resp_ingress_count 1
bigstart restart asm
Component: Local Traffic Manager
Symptoms:
The GUI and TMSH report an error message if a symlink is present, and the workspace does not run. The error appears similar to the following:
General database error retrieving information.
General error: 01070711:3: boost::filesystem::status: Permission denied: "/var/ilx/workspaces/Common/test_links1/tmp_file" in statement [SELECT COUNT(*) FROM dev_workspace WHERE name LIKE '%'].
Conditions:
-- An ILX workspace is in the configuration.
-- The workspace contains a symlink.
-- Install the relevant rpm package with --no-bin-links (e.g., npm install <package-name> --no-bin-links).
Impact:
The ILX module is not accessible via the GUI, and the workspace with the symlink cannot be run.
Workaround:
1. Remove the symlink.
2. Copy the file into the workspace.
Component: Local Traffic Manager
Symptoms:
TMM could crash with core when HTTP Security Profile (Protocol Security, PSM) is on the Virtual Server, and using an iRule with either the HTTP::redirect or HTTP::respond commands, together with HTTP::disable on the same event. This is normally an incorrectly written iRule, but TMM crashes in this case.
Conditions:
-- HTTP Security Profile is used.
-- iRules contain HTTP::disable command and either HTTP::redirect or HTTP::respond on the same event.
Impact:
Traffic disrupted while tmm restarts.
Workaround:
Edit the iRules to prevent calling HTTP::disable together with HTTP::respond or HTTP::redirect.
Component: Local Traffic Manager
Symptoms:
TMM NDP entries show correct info with neighbor discovery protocol (NDP) resolved but ping6 fails
Conditions:
This occurs only on a cluster setup. Other conditions that cause the issue are unknown.
Impact:
Ping6 fails for that address.
Workaround:
None.
Component: TMOS
Symptoms:
After an upgrade to v14.1.0 or higher, swap memory may not be mounted. TMM or other host processes may restart due to lack of memory.
Conditions:
-- System is upgraded to v14.1.0 or above.
-- System has RAID storage.
Impact:
May lead to low or out-of-memory condition. The Linux oom killer may terminate processes, possibly affecting service.
Typically management activities may be impacted, for example, a sluggish GUI (config utility) or tmsh sessions.
Workaround:
Mount the swap volume with correct ID representing the swap device.
Perform the following steps on the system after booting into the affected software version:
1. Get the correct ID (RAID device number (/dev/md<number>)):
blkid | grep swap
Note: If there is no RAID device number, perform the procedure detailed in the following section.
2. Check the device or UUID representing swap in /etc/fstab.
3. If swap is not represented with the correct ID, modify the /etc/fstab swap entry to point to the correct device.
4. Enable the swap:
swapon -a
5. Check swap volume size:
swapon -s
If the blkid command shows there is no UUID associated with the swap RAID device, use the following procedure:
1. Generate a random UUID:
uuidgen
2. Make sure swap is turned off:
swapoff -a
3. Recreate the swap partition with UUID generated in step 1:
mkswap -U <uuid_from_step_1> <raid_device_from_step_1>
4. Run blkid again to make sure that you now have a UUID associated with the raid device:
blkid | grep swap
5. edit fstab and find the line
<old_value> swap swap defaults 0 0
6. Replace the old value, whether it was an incorrect UUID or a device name, with the UUID generated in step 1, for example:
UUID=8b35b30b-1076-42bb-8d3f-02acd494f2c8 swap swap defaults 0 0
Component: TMOS
Symptoms:
Parts of audit logs are found also in other logs like /var/log/user.log and /var/log/messages.
Conditions:
-- When audit log message strings are long.
Impact:
Log messages are duplicated. There is no indication of system functionality, and you can safely ignore them.
Workaround:
Modify the syslog-ng maximum length of incoming log messages from 8192 to 16384 bytes:
tmsh modify sys syslog include "options { log-msg-size(16384); };"
Component: Global Traffic Manager (DNS)
Symptoms:
BIG-IP system may lose DNSSEC keys if the key generator is changed from rolling keys to static keys
Conditions:
DNSSEC key generation is changed from rolling to static.
Impact:
DNSSEC keys may be lost.
Workaround:
None.
Component: Global Traffic Manager (DNS)
Symptoms:
BIG-IP GTM might not always sync a generation of a DNSSEC key from its partner.
Conditions:
Generating a DNSSEC key.
Note: This is an intermittent issue.
Impact:
DNSSEC keys may not be synced.
Workaround:
None.
Component: Device Management
Symptoms:
-- The 'top' command shows Java and mcpd becoming CPU intensive.
-- /var/log/audit shows many 'modify { user_role_partition { user_role_partition_user ...'
-- /var/log/restjavad-audit.0.log shows many REST API calls to 'http://localhost:8100/mgmt/shared/gossip' from the peer.
Conditions:
-- Create a new admin user with bash access on a device.
Impact:
High CPU usage (Java and mcpd) on control and analysis plane.
Workaround:
To work around this issue, follow these steps:
1. Sync from the device where the user was created.
2. Run the following command on all devices:
tmsh restart sys service restjavad
Although Java and mcpd will still show high CPU usage even after restart, waiting a few minutes enables the processes to return to normal.
Component: Protocol Inspection
Symptoms:
When saving the configuration to a UCS file, the process tries save the IPS learning information stored in the MySQL database.
MySQL runs only when particular modules are provisioned. If MySQL was previously running as a result of different provisioning, but is not currently running, saving the configuration to a UCS file succeeds, but the system reports a spurious message during the operation:
Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock.
Conditions:
-- Saving the configuration to a UCS file.
-- BIG-IP system provisioning only includes modules that do not require MySQL. These modules may include:
+ LTM
+ FPS
+ GTM (DNS)
+ SWG
+ iLX
+ SSLo
-- BIG-IP system was previously provisioned with a module that starts MySQL, which results in the creation of the file /var/db/mysqlpw. These modules may include:
+ APM
+ ASM
+ AVR
+ PEM
+ AFM
+ vCMP
Impact:
The error message is cosmetic and has no impact on the UCS save process.
Workaround:
None.
Component: Access Policy Manager
Symptoms:
Several APM/Access BigDB keys are obsolete and should be removed as they only add confusion
Conditions:
--BigIp is UP and Running
Impact:
Though those keys are not being used they create confusion as a placeholder
Workaround:
Remove those keys from BigDB and control plane side as those are not being used. But don't remove the keys which has still dependancies with other modules and also don/'t remove those keys used in upgrade
Component: Global Traffic Manager (DNS)
Symptoms:
LTM configuration does not auto-discover GTM-configured virtual servers.
Conditions:
-- GTM is deprovisioned on one or more GTM sync group members, or the sync group is reconfigured on one or more members.
-- Those devices remain present in the GTM configuration as 'gtm server' objects.
-- iQuery is connected to those members.
Impact:
Virtual servers are not discovered or added automatically.
Workaround:
You can use either of the following workarounds:
-- Manually add the desired GTM server virtual servers.
-- Delete the 'gtm server' objects that represent the devices that are no longer part of the GTM sync group. These can then be recreated if the devices are operating as LTM-configured devices.
Component: Local Traffic Manager
Symptoms:
TCP Verified-Accept with proxy-mss enabled does not honor the route-metrics cache when sending the SYN to the server
Conditions:
-- TCP Verified-Accept option is used.
-- The proxy-mss is enabled.
-- The route-metrics cache entry is enabled.
Impact:
Route-metrics cache entry does not get used.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
You see 'SSL handshake timeout' error messages in LTM log, and high availability (HA) system performance becomes degraded.
Conditions:
This might occur in either of the following scenarios:
Scenario 1
-- Manual sync operations are performed during while traffic is being passed.
-- SSL Connection mirroring is enabled.
Scenario 2
-- Saving configuration on an HA Standby node during while traffic is being passed.
-- SSL Connection mirroring is enabled.
Impact:
-- In Scenario 1, the sync operations causes the session cache to be out-of-sync between active and standby nodes.
-- In Scenario 2, the save operation clears the session cache on the standby node. As a result, the session cache might be out-of-sync between active and standby nodes.
In either Scenario:
-- SSL Connection mirroring fails and posts the timeout message.
-- The HA system performance becomes degraded due to SSL connection timeout.
Workaround:
-- Disable SSL session caching by setting 'Cache Size' in the client SSL profile option to 0.
-- Set device management sync type to Automatic with incremental sync.
Component: TMOS
Symptoms:
Unit suddenly stops passing traffic. You might see errors similar to the following:
err mcpd[15230]: 01070596:3: An unexpected failure has occurred, TAP creation failed (tmm): Permission denied - net/validation/routing.cpp, line 168, exiting...
Conditions:
This might occur when when /var/log is full and then you remove big logs.
Impact:
The mcpd process restarts continuously. This occurs because tmm blocks mcpd from restarting after /var/log fills up.
Workaround:
Fix the logs and reboot the BIG-IP system.
Component: TMOS
Symptoms:
The snmpd configuration parameters can cause net-snmp to issue a warning about deprecated syntax.
Conditions:
The use of a sys snmp command similar to the following to modify the snmpd.conf file:
sys snmp v2-traps { TRAP1 { host 1.2.3.4 community somestring } }
Impact:
net-snmp issues a warning that the syntax has been deprecated and reports a warning message in the log.
Workaround:
None.
Component: TMOS
Symptoms:
CPU usage statistics reported for Control and Analysis planes are not described properly for single-core vCMP guests.
Conditions:
A vCMP guest with a single core.
Impact:
CPU usage statistics report 0 Control Plane cores and 1 Analysis Plane core.
Workaround:
On a single core, two hyperthread vCMP guest, one hyperthread/CPU is dedicated to Data Plane while the other is dedicated to Control and Analysis Plane. All statistics attributed to the Analysis Plane in this CPU configuration are in fact the aggregate of Control Plane and Analysis Plane.
Component: TMOS
Symptoms:
External link-state advertisement (LSA) update does not trigger OSPF ASE route calculation, resulting in delay for route state changes from external LSA.
Conditions:
-- OSPF enabled.
-- More than 20 updated external LSA.
-- No updated router and network LSA.
Impact:
Delay of route update from external LSA.
Workaround:
Manually clear ip ospf process.
Component: Application Security Manager
Symptoms:
An unescaped slash in RE2 keyword in a user-defined signature caused a REST PATCH to the signature to have no effect.
Conditions:
A user-defined signature has an unescaped slash in RE2 keyword.
Impact:
REST PATCH to update the user-defined signature has no effect.
Workaround:
The slash in the signature keyword must be escaped by backslash.
Component: TMOS
Symptoms:
Guestagentd periodically logs the following REST error message for each secondary slot in /var/log/ltm:
Rest request failed{"code":502."message":"This is a non-primary slot on the Viprion. Please access this device through the cluster address.","restOperationId":6410038,"kind":":resterrorresponse"}
Conditions:
Upgrade a vCMP guest from pre-13.1.x to a 13.1.x or later version.
Impact:
There is stale stat information for vCMP guests running on secondary slots.
Workaround:
Create a Log Filter with no publisher on the vCMP guest to discard the specific error message:
sys log-config filter Filter_RestError {
level info
message-id 01810007
source guestagentd
Component: TMOS
Symptoms:
RADIUS authentication can only have an initial service type of 'authenticate only'.
Conditions:
This is encountered when configuring RADIUS authentication via the GUI.
Impact:
If you change the Service Type to anything except Authenticate Only (default), Authentication creation fails, and the following error appears in /var/log/webui.log:
01020066:3: The requested RADIUS Authentication Configuration (/Common/system-auth) already exists in partition Common.
Workaround:
After configuring RADIUS authentication with 'authenticate only' as the service type, go back and change the service type to the desired option.
Component: TMOS
Symptoms:
Monitor 'Instances' tab shows incorrect pools if the same members are used in other pools.
Conditions:
Steps to Reproduce:
1. Create custom monitor or use system default.
2. Assign that monitor to a test pool.
3. Navigate to Local Traffic :: Monitors, click the test monitor, then select the Instances tab.
Impact:
The test pool is displayed, as well any other pools that use the same member or members (but with other monitors assigned).
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
For Thales:
The ltm/log shows SSL handshake failures with similar lines (this is for Diffie-Hellman Key Exchange):
-- warning bigip1 tmm1[28813] 01260013 SSL Handshake failed for TCP 192.0.2.1:5106 -> 192.0.2.200:5607
-- warning bigip1 tmm1[28813] 01260009 Connection error: ssl_hs_vfy_sign_srvkeyxchg:13583: sign_srvkeyxchg (80)
-- debug bigip1 tmm1[28813] 01260036 FIPS acceleration device error: fips_poll_completed_reqs: req: 4 status: 0x1 : Cancel
-- err bigip1 pkcs11d[26259] 01680002 Key table lookup failed. error.
After enabling pkcs11d debug, the pkcs11d.debug log shows:
-- 2019-10-03 11:21:50 [6399] t00075a9a462b0000: pkcs11: 000008D9 D obj_match_attribute class CKO_PRIVATE_KEY attribute CKA_CLASS
-- 2019-10-03 11:21:50 [6399] t00075a9a462b0000: pkcs11: 000008D9 D obj_match_attribute type CKA_CLASS matches
-- 2019-10-03 11:21:50 [6399] t00075a9a462b0000: pkcs11: 000008D9 D obj_match_attribute class CKO_PRIVATE_KEY attribute CKA_ID
-- 2019-10-03 11:21:50 [6399] t00075a9a462b0000: pkcs11: 000008D9 D obj_match_attribute type CKA_ID does not match <===
For Safenet:
-- warning tmm1[17495]: 01260009:4: Connection error: ssl_hs_vfy_sign_srvkeyxchg:13544: sign_srvkeyxchg (80)
-- warning tmm1[17495]: 01260013:4: SSL Handshake failed for TCP 10.1.1.11:6009 -> 10.1.1.201:443
-- err pkcs11d[5856]: 01680002:3: Key table lookup failed. error.
Conditions:
1. Keys were created on earlier versions of BIG-IP software, no matter if using tmsh (Safenet) or using fipskey.nethsm (Thales, Safenet) and the device was upgraded to 14.1.0 or later.
2. Keys were created on BIG-IP v14.1.0 or later directly, using fipskey.nethsm (Thales). For Safenet, fipskey.nethsm was deprecated in 14.0.0.
Impact:
SSL handshake failures.
Workaround:
There are two workarounds:
-- Re-create the keys using tmsh command.
IMPORTANT: This workaround is suitable for deployments that are new and not in production.
-- Re-import the keys from nethsm using:
tmsh install sys crypto key <key_label> from-nethsm
You can find the key_label here:
-- The rightmost string in the output of the Thales command:
nfkminfo -l
-- The string after label= in the 'cmu list' command for Safenet.
Component: Local Traffic Manager
Symptoms:
Which actions trigger in a first-match policy should depend on the ordinal of their rule. Sometimes, this does not work correctly.
Conditions:
The conditions under which this occurs are not known.
Impact:
LTM policy rules do not execute in the expected order.
Workaround:
It may be possible to re-arrange the rules to avoid the incorrect action execution.
Component: Application Security Manager
Symptoms:
The online help of Proactive Bot Defense within the DoS profile shows the following under the 'Cross-Domain Requests' section:
Allow configured domains; validate in bulk: ... We recommend this option if your web site has many cross-domain resources.
Allow configured domains; validate upon request: ... We recommend this option if your web site does not have many cross-domain resources.
The recommendation is actually the reverse: for many cross-domain resources, it is better to use 'validate upon request'.
Conditions:
Application has multiple cross-domain resources.
Impact:
Confusing documentation. The recommendation is actually the reverse: for many cross-domain resources, it is better to use 'validate upon request'.
Workaround:
For many cross-domain resources, it is better to use 'validate upon request'.
Component: Local Traffic Manager
Symptoms:
When two or more database monitors (MSSQL, MySQL, PostgreSQL, Oracle) with identical 'send' and 'recv' strings are configured and applied to different pools (with at least one pool member in each), the monitor status of some pool members may not be updated accurately.
Other parameters of the affected monitors that differ (such as 'recv row' or 'recv column' indicating where the specified 'recv' string should be found in the result set) may cause pool members using one of the affected monitors to connect to the same database to be marked UP, while pool members using another affected monitor may be marked DOWN.
As a result of this issue, pool members that should be marked UP or DOWN by the configured monitor may instead be marked according to another affected monitor's configuration, resulting in the affected pool members being intermittently marked with an incorrect state.
After the next monitor ping interval, affected pool members members may be marked with the correct state.
Conditions:
This may occur when multiple database monitors (MSSQL, MySQL, PostgreSQL, Oracle) are configured with identical 'send' and 'recv' parameters, and applied to different pools/members.
For example:
ltm monitor mysql mysql_monitor1 {
recv none
send "select version();"
ltm monitor mysql mysql_monitor2 {
recv none
send "select version();"
Impact:
Monitored pool members using a database monitor (MSSQL, MySQL, PostgreSQL, Oracle) randomly go offline/online.
Workaround:
To avoid this issue, configure each database monitor with values that make the combined parameters unique by changing either the 'send' or the 'recv' parameters, or both.
For example:
ltm monitor mysql mysql_monitor1 {
recv none
send "select version();"
ltm monitor mysql mysql_monitor2 {
recv 5.7
send "select version();"
Component: TMOS
Symptoms:
When creating a new rule or modifying an existing rule in a LTM Policy using the WebUI, the operation fails and an error similar to the following example is returned:
Transaction failed:010715bd:3: The parent folder is owned by application service (/Common/MyPolicy.app/MyPolicy), the object ownership cannot be changed to ().
Conditions:
-- The LTM Policy belongs to an Application Service (iApp).
-- The modification is attempted via the WebUI.
Impact:
Unable to make changes to existing LTM Policies.
Workaround:
Use the tmsh utility to make the necessary modifications to the LTM Policy. For example, the following command modifies an existing rule:
tmsh modify ltm policy myapp.app/Drafts/myapp_l7policy rules modify { 0 { conditions modify { 0 { http-method equals values { GET POST } } } } }
Component: Application Security Manager
Symptoms:
When Bot Defense profile is used and has Browser Verification enabled to either Verify Before Access, or Verify After Access, Microsoft Internet Explorer v11 (IE11) Browsers may display the following errors in the browser console:
HTML1512: Unmatched end tag.
a.html (26,1)
HTML1514: Extra "<body>" tag found. Only one "<body>" tag should exist per document.
a.html (28,1)
Conditions:
-- Bot Defense profile is enabled with Browser Verification.
-- End user clients are using the IE11 browser.
Impact:
Cosmetic error messages appear in an end user's browser console.
Workaround:
In order to work around this issue, inject the scripts to the '<body>' tag instead of the '<head>' tag. This can be done using this tmsh command:
tmsh mod sys db dosl7.parse_html_inject_tags value after,body,before,/body`
Component: Global Traffic Manager (DNS)
Symptoms:
Attempt to delete a DNS Validating Resolver cache record from the 'Key' cache does not remove the record. Also displays a negative TTL for that record.
tmm crash
Conditions:
-- Populate the DNS Validating Resolver Cache.
-- Attempt to delete a record from the 'Key' cache.
Impact:
Undesired behavior due to records not being deleted as instructed. Also negative TTL.
Workaround:
The only workaround is to restart tmm to generate a completely empty DNS cache. Traffic disrupted while tmm restarts.
Component: TMOS
Symptoms:
On vCMP guests, logs of 'MSIX is not supported' messages apppear in /var/log/tmm.
Conditions:
This occurs only on vCMP guests.
Impact:
MSIX is not supported on vCMP guests, but system operation and traffic passing are not impacted otherwise.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
When using FQDN nodes and pool members, ephemeral pool members may not be created as expected immediately after a configuration-load or BIG-IP system reboot operation.
Conditions:
This may occur on affected BIG-IP versions when:
-- Multiple FQDN names (configured for FQDN nodes/pool members) resolve to the same IP address.
-- DNS queries to resolve these FQDN names occur almost simultaneously.
The occurrence of this issue is very sensitive to timing conditions, and is more likely to occur when there are larger numbers of FQDN names resolving to a common IP address.
Impact:
When this issue occurs, some subset of ephemeral pool members may not be created as expected.
As a result, some pools may not have any active pool members, and do not pass traffic.
This issue, when it occurs, may persist until the next DNS queries occur for each FQDN name, at which point the missing ephemeral pool members are typically created as expected. Using the default FQDN interval value of 3600 seconds, such downtime lasts approximately one hour.
Workaround:
To minimize the duration of time when pools may be missing ephemeral pool members, configure a shorter FQDN interval value for the FQDN nodes:
tmsh mod ltm node fqdn-node-name { fqdn { interval ## } }
Where ## is the desired number of seconds between successive DNS queries to resolve the configure FQDN name.
Component: TMOS
Symptoms:
The BIG-IP system may fail source translation for connections matching a virtual server that has connection mirroring enabled and source port selection set to 'preserve strict'.
Conditions:
Connections match a virtual server that has following settings:
- Connection mirroring is enabled.
- Source Port set to 'Preserve Strict'.
In addition, CMP hash selection (DAG mode) on the corresponding VLANs is set to 'Default DAG'.
Impact:
Source translation may fail on BIG-IP system, leading to client connection failures.
Workaround:
You can try either of the following:
-- Do not use the Source Port setting of 'Preserve Strict'.
-- Disable connection mirroring on the virtual server.
Component: Local Traffic Manager
Symptoms:
When the tmm Nitrox 3 queue stuck problem is encountered, the Nitrox 3 code tries to log the instruction/request, but it may fail due to SELinux permissions error.
The system posts messages in /var/log/ltm similar to the following:
-- crit tmm1[21300]: 01010025:2: Device error: n3-compress0 Nitrox 3, Hang Detected: compression device was reset (pci 00:09.7, discarded 54).
-- crit tmm1[21300]: 01010025:2: Device error: n3-compress0 Failed to open instruction log file '/shared/nitroxdiag/instrlog/tmm01_00:09.7_inst.log' err=2.
Conditions:
-- tmm Nitrox 3 queue stuck problem is encountered.
-- The Nitrox 3 code tries to log the instruction/request.
Impact:
Error messages occur, and the tmm Nitrox 3 code cannot log the instruction/request.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
After an HTTPS monitor fails, it never resumes probing. No ClientHello is sent, just 3WHS and then 4-way closure. The pool member remains down.
Conditions:
-- The cipherlist for the monitor is not using TLSv1 (e.g., contains -TLSv1 or !TLSv1).
-- The pool member is marked down.
Impact:
Services are not automatically restored by the health monitor.
Workaround:
-- To restore the state of the member, remove it and add it back to the pool.
-- Remove !TLSv1 and -TLSv1 from the cipher string, if possible.
Component: TMOS
Symptoms:
After an mcpd daemon restart, sometimes (in ~30% of cases) ZebOS is missing some of kernel routes (virtual addresses).
One of the most common scenario is a device reboot.
Conditions:
-- Dynamic routing is configured.
-- Virtual address is created and Route Advertisement is configured:
imish -e 'sh ip route kernel'
-- mcpd daemon is restarted or device is rebooted.
Impact:
The kernel route (virtual address) is not added to the ZebOS routing table and cannot be advertised.
Workaround:
There are several workarounds; here are two:
-- Restart the tmrouted daemon:
bigstart restart tmrouted
-- Recreate the affected virtual address.
Component: Local Traffic Manager
Symptoms:
Unexpected or improper pool/node member status.
Conditions:
The BIG-IP system receives the ICMP rejection code as icmp-net/host-unreachable.
Impact:
The monitor might consider a server available when some type of ICMP rejection has been received that is not port unreachable.
Workaround:
You can use either of the following workarounds:
-- Use UDP monitors configured with a receive string.
-- Do not use UDP monitors.
Component: Local Traffic Manager
Symptoms:
The UDP / DNS monitor marks nodes down.
Conditions:
-- UDP or DNS monitor configured.
-- Interval is multiple of timeout.
-- The response is delayed by over one interval.
Impact:
Pool member is marked down.
Workaround:
Increase the interval to be greater than the response time of the server.
Component: Service Provider
Symptoms:
From UI, when navigate to the following path:
Statistics ›› Module Statistics : Local Traffic ›› Profiles Summary : Diameter Router.
The stat for 'Current number of dynamic routes' does not appear.
Conditions:
Any condition
Impact:
Unable to view dynamic routes statistics in the GUI
Workaround:
Look at the statistics from tmsh
Component: Local Traffic Manager
Symptoms:
In an iRule such as 'string first $string1 $string2' returns incorrect results when $string2 contains a null byte and $string1 is not found within $string2. Performing the same search in tclsh, the expected -1 (not found) result is returned.
Conditions:
-- 'string first $string1 $string2' iRule.
-- string2 in an iRule contains a null byte.
Impact:
Operation does not return the expected -1 (not found) result, but instead returns an unexpected, random result.
Workaround:
None.
Component: TMOS
Symptoms:
The install ISO does not boot on BIG-IP Virtual Edition (VE).
Conditions:
Attempting to boot a BIG-IP VE from a virtual DVD-ROM drive loaded with an affected ISO file.
Impact:
The system does not fully boot and hangs, preventing you from performing an installation or using the live environment for other recovery purposes.
Workaround:
To work around this issue, boot the BIG-IP VE from an ISO file earlier than 14.1.0. If necessary, install that version, and then upgrade to 14.1.0 using the live installer.
Component: Application Security Manager
Symptoms:
False-positive JSON max array length violation is reported.
Conditions:
-- JSON profile is used.
-- The violation is coming for non-array under certain conditions.
Impact:
The system reports a false-positive violation.
Workaround:
None.
Component: TMOS
Symptoms:
Configurations that previously used TACACS for authentication in order to make REST requests are no longer required to use a token for remote authentication. You can simply use username and password via basic authentication.
Conditions:
Use of remote authentication using TACACS.
Impact:
iControl REST traffic using Basic authentication with valid credentials succeeds, when it should not.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
When plain literal IP address and mask are used in IP::addr command, the validation fails if the mask is single digit.
Conditions:
The address mask is single digit.
Impact:
Validation fails.
Workaround:
Assign address/mask to a variable and use the variable in the command.
Component: Service Provider
Symptoms:
Some iRule commands (such as relate_server) fail when used in conjunction with Message Routing Protocol (MRP) SIP configurations using message routing transport.
Conditions:
-- MRP SIP configuration uses transport-config.
-- iRule command 'relate_server' is configured on the corresponding virtual server.
Impact:
iRule commands such as relate_server cannot be used with MRF SIP.
Workaround:
None.
Component: Access Policy Manager
Symptoms:
APMD reports 'too many open files' error when reading HTTP requests:
-- err apmd[15293]: 01490000:3: HTTPParser.cpp func: "readFromSocket()" line: 113 Msg: epoll_create() failed [Too many open files].
-- err apmd[15293]: 01490000:3: ApmD.cpp func: "process_apd_request()" line: 1801 Msg: Error 3 reading/parsing response from socket 1498. strerror: Too many open files, queue size 0, time since accept
There are file descriptor dumps in /var/log/apm showing many deleted files with name krb5_RCXXXXXX:
-- err apmd[15293]: 01490264:3: 1492 (/shared/tmp/krb5_RCx8EN5y (deleted)) : cloexec, Fflags[0x8002], read-write
-- err apmd[15293]: 01490264:3: 1493 (/shared/tmp/krb5_RCnHclFz (deleted)) : cloexec, Fflags[0x8002], read-write
-- err apmd[15293]: 01490264:3: 1494 (/shared/tmp/krb5_RCKGW8ia (deleted)) : cloexec, Fflags[0x8002], read-write
Conditions:
This failure may happen if the access policy uses Kerberos authentication, Active Directory authentication, or Active Directory query. The conditions under which the Kerberos replay cache leaks is unknown.
Impact:
APM end users experience intermittent log on issues.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
Translate-address disabled on an HTTP/2 virtual server is ignored.
Conditions:
-- HTTP/2 virtual server configured.
-- Translate-address disabled.
Impact:
The traffic is still translated to the destination address to the pool member.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
If there are HTTPS monitor objects that were created using BIG-IP software v12.x, when the BIG-IP is upgraded directly to v14.0.0 or later, the operation automatically creates server SSL profiles for the HTTPS monitors as needed. Those server SSL profile objects do not have 'no-tlsv1.3' included in their 'options' configuration.
Conditions:
-- Having HTTPS monitors configured in v12.x before upgrading.
-- Directly upgrading from v12.x to v14.0.0 or later
Impact:
TLSv1.3 gets enabled on the server SSL profiles.
Workaround:
-- To avoid this issue, upgrade from v12.x to v13.x, and then upgrade to v14.0.0 or later
-- To mitigate this issue, modify the affected profile to disable TLSv1.3.
Component: TMOS
Symptoms:
It is possible to configure two virtual servers with the same address, port, and route domain, and have them overlap only in VLANs. MCP does not detect the overlap.
Errors like this may be seen in the ltm log:
err tmm1[29243]: 01010009:3: Failed to bind to address
Conditions:
Two (or more) virtual servers with the same address, port, and route domain, and have them overlap only in VLANs
Impact:
Traffic does not get routed properly.
Workaround:
There is no workaround other than ensuring that virtual servers that have the same address, port, and route domain have no overlap of VLANs.
Component: Local Traffic Manager
Symptoms:
LTM logs are filled with error messages when creating/deleting virtual wire config.
Conditions:
Virtual wire is created and then deleted.
Impact:
Error messages are getting logged to ltm.
Component: TMOS
Symptoms:
When manually copying a large config file and running 'tmsh load sys config' on specific hardware BIG-IP platforms, multiple cores may be observed from different daemons.
Conditions:
This has been observed on i4800 platforms when the 'management' provisioning (corresponding to the provision.extramb DB key) is set to 500 MB or less.
Impact:
The mcp daemon may core and all daemons on the BIG-IP system may be restarted.
Workaround:
Set db key 'provision.extramb' to 1024 or greater.
Component: TMOS
Symptoms:
When running qkview with the -h option to obtain help, the -s (size) option is incorrectly rendered.
It should read:
[ -s <max file size> range:0-104857600 Bytes ]
Conditions:
-- Running qkview -h.
-- Viewing the -s (size) option help.
Impact:
The measurement size, bytes, is missing, which might result in confusion.
Workaround:
Use the -s option as normal, but be advised that the number should be in bytes, and that the maximum number is 104857600.
Component: Service Provider
Symptoms:
Under certain conditions the wrong self-IP can be selected as a source address for connections from an Internal Virtual Server to remote servers.
Conditions:
- Using an Internal Virtual Server (IVS).
- The VLAN being used to connect from the IVS to the server does not have a floating self-IP configured.
- At least one other VLAN has a floating self-IP configured.
- The primary virtual server that connects to the IVS is using SNAT automap.
Impact:
IVS traffic might not be routed properly.
Workaround:
- Configure a floating self-IP on the IVS server side VLAN.
- Use a SNAT pool instead of automap.
Component: TMOS
Symptoms:
REST PUT fails to update the object description when proxy-ca-cert and proxy-ca-key are not configured, and triggers an error:
SSL forward proxy RSA CA key is missing.
Conditions:
Issue is seen only with REST, and when proxy-ca-cert and proxy-ca-key are not configured.
Impact:
REST cannot be used to update/modify the description.
Workaround:
You can use TMSH to update/modify the description, even if proxy-ca-cert and proxy-ca-key are not configured.
Component: TMOS
Symptoms:
South Sudan, Sint Maarten, Curacao countries are missing from the region list.
Conditions:
-- Creating a GTM region record.
-- Create a GTM any region of Country South Sudan
Impact:
Cannot select South Sudan county from GTM country list.
Workaround:
Component: Advanced Firewall Manager
Symptoms:
In Security -> Network Firewall -> Active Rules, When selecting a Virtual Server context, The policy associated with the Virtual Server is listed, but not the rules within that policy
Conditions:
Applies to Virtual Servers policies
Impact:
User cannot manage rules from Active Rules page. This is UI display only issue and does not affect functionality.
Workaround:
In Security -> Options -> Network Firewall -> Firewall Options:
Disable Inline Rules. This will revert back to the legacy edirot and will display the policy details
Component: Service Provider
Symptoms:
Certain irregular SIP message patterns may produce an erroneous curr_pending_calls value that can drop below zero and underflow.
Conditions:
Uncommon message flows like re-transmitted INVITE or OK responses can trigger the issue, which may be brought about at times by lost packets when using UDP.
Impact:
SIP curr_pending_calls may show incorrect values.
Component: Access Policy Manager
Symptoms:
The discovery task runs but does not apply the 'Access Access Policy' for the access policy for which the Provider is configured.
Conditions:
-- Auto-discovery is enabled for a provider.
-- Discovery occurs.
Impact:
The Access Policy is not applied after successful auto-discovery. The policy must be applied manually.
Workaround:
Apply the Access Policy manually after auto-discovery.
Component: Application Security Manager
Symptoms:
When Referer header contains a backslash character ('\') in query string portion, 'IIS backslashes' evasion technique violation is raised.
Conditions:
-- 'Url Normalization' is turned on and 'Evasion Techniques Violations' is enabled.
-- Referer header contains a backslash character ('\') in query string part.
Impact:
False positive evasion technique violation is raised for Referer header.
Workaround:
Turn off 'Url Normalization' on the 'Normalization Settings' section of the 'referer' property on the HTTP Header Properties screen.
Component: TMOS
Symptoms:
The SFP interfaces do not come up or flap up and down repeatedly on BIG-IP 2000/4000 on boot up when both SFP interfaces are populated.
When interface flaps state changes such as those below are logged in ltm log:
info pfmand[PID]: 01660009:6: Link: 2.1 is UP
info pfmand[PID]: 01660009:6: Link: 2.1 is DOWN
Conditions:
Both SFP interfaces, 2.1 and 2.2, on BIG-IP 2000/4000 are populated.
This is typically observed after an upgrade to an affected version.
Impact:
Traffic cannot be sent/received from these interfaces.
Workaround:
None.
Component: TMOS
Symptoms:
If, during qkview operation's exit stage, it is interrupted (with Ctrl-C for example), it produces a core file.
Conditions:
-- Qkview is exiting.
-- The qkview operation receives an interrupt.
Impact:
A core file is produced.
Workaround:
When closing qkview, or if it is closing, do not interrupt it; wait for it to exit.
Component: TMOS
Symptoms:
Unit goes offline after UCS install. Secondary blades go offline. This lasts about a minute, and then the system goes back online.
Conditions:
After UCS install.
Impact:
-- Limited high availability (HA) capabilities (failover, sync, mirroring, etc.).
-- Cluster reduced to a single blade immediately after UCS install, which might impact performance.
Workaround:
None.
Component: Global Traffic Manager (DNS)
Symptoms:
Rolling DNSSEC Keys may stop generating when a BIG-IP system restart.
Conditions:
BIG-IP system gets restarted by calling 'bigstart restart' command.
Impact:
Rolling DNSSEC keys can stop generating.
Workaround:
None.
Component: Access Policy Manager
Symptoms:
Config sync operation fails for APM policy when policy item of same name points to different agent on source and target
The system posts errors similar to the following:
Sync error on rfang-vemgmt.lab.labnet.com: Load failed from /Common/rfang-ve-3mgmt.lab.labnet.com 01070734:3: Configuration error: DB validation exception, unique constraint violation on table (access_policy_item_agent) object ID (/Common/resm_act_message_box_1 /Common/resm_act_message_box_ag_1). A duplicate value was received for a non-primary key unique index field. DB exception text (Cannot update_indexes/checkpoint DB object, class:access_policy_item_agent status:13)"
Conditions:
This occurs in the following scenario:
1. Configure a failover device group containing two BIG-IP systems.
2. Create an APM access profile on one unit.
+ Launch VPE for the policy.
+ Add a macro.
+ In macro add an agent, e.g., Message box.
+ Add macro to the main policy.
3. Initiate config sync to another device.
4. On one BIG-IP system, add another Message box agent using the same macro. On the other BIG-IP system, make a copy of the access profile.
5. On either BIG-IP system, initiate another config sync operation.
Impact:
Unable to sync configuration in a failover device group.
Workaround:
You can work around this using the following procedure:
1. On the device receiving the config sync, delete the APM policies that contain the referenced APM policy items.
2. Perform an overwrite-config-sync operation from the sending device to this device.
Component: Application Visibility and Reporting
Symptoms:
When a pool member is defined with port any(0), calling the GetPoolMember() function, gives an incorrect error message that the pool member was not found.
Conditions:
Pool member with port any(0)
Impact:
Wrong error message printed to avrd.log
Component: Local Traffic Manager
Symptoms:
FastL4 does not re-assemble fragments by default, but on a system with software-enabled TSO (sys db tm.tcpsegmentationoffload value disable), those fragments are erroneously re-segmented.
Conditions:
The behavior is encountered on BIG-IP Virtual Edition when setting sys db tm.tcpsegmentationoffload value disable, but does not cause a tmm core on Virtual Edition.
Impact:
Already-fragmented traffic is fragmented again.
Workaround:
Component: TMOS
Symptoms:
If the BIG-IP system's WebUI is accessed via certain special proxy solutions, logging on to the system may fail.
Conditions:
This issue is known to happen with special proxy solutions that do one of the following things:
- Remove the Referer header.
- Modify the HTTP request in such a way that the Referer and Host headers no longer tally with one another.
Impact:
Users cannot log on to the BIG-IP system's WebUI.
Workaround:
As a workaround, you can do any of the following things:
- Access the BIG-IP system's WebUI directly (i.e., bypassing the problematic proxy solution).
- Modify the proxy solution so that it does not remove the Referer header (this is only viable if the proxy does not alter the Host header).
- Modify the proxy solution so that it inserts compatible Referer and Host headers.
Component: Local Traffic Manager
Symptoms:
The WebSocket feature has statistics that records the number of each type of frame seen. These statistics do not differentiate between client and server sides.
Conditions:
The WebSocket profile is used to add WebSocket protocol parsing.
Impact:
WebSocket Traffic Statistics may be misleading
Workaround:
None.
Component: TMOS
Symptoms:
Configuring multiple SR-IOV interfaces into a trunk does not function correctly when running BIG-IP as a guest under VMware ESXi. The interface will show as uninitialized.
Conditions:
A system that passes SR-IOV virtual functions directly to a BIG-IP guest when running on VMware ESXi.
Impact:
The trunk will fail to initialize.
Workaround:
None.
Component: TMOS
Symptoms:
The network map loads slowly when it contains lots of objects.
Conditions:
Load the network map in a configuration that contains 1000 or more objects.
Impact:
The network map loads very slowly.
Workaround:
None.
Component: TMOS
Symptoms:
TMM crashes during normal operation.
Conditions:
-- This can occur while passing normal traffic.
-- In this instance, APM and LTM are configured.
Impact:
Traffic disrupted while tmm restarts.
Workaround:
None.
Component: TMOS
Symptoms:
TACACS authentication does not support IPv6 address for the authentication server, but both GUI and TMSH allow IPv6 addresses to be configured for TACACS. Such configurations may result in failed logins with messages in /var/log/secure like
Aug 8 10:47:39 gtm-13108-174 err httpd[5948]: pam_tacplus: skip invalid server: 2001::1001:1001 (invalid port: no digits)
Conditions:
Use the GUI or TMSH to create or modify a TACACS server
Impact:
Remote authentication will fail unless a second server is configured with IPv4 address.
Workaround:
Do not configure IPv6 address for TACACS server
Component: Global Traffic Manager (DNS)
Symptoms:
Wide IP persistence is not working. Previous Wide IP persistence records are cleared.
Conditions:
This occurs when the Wide IP Persistence TTL plus the persist-record creation time is greater than 4294967295.
Impact:
Wide IP persistence does not work.
Workaround:
There is no workaround other than not setting Wide IP Persistence TTL to a number greater than 4294967295.
Component: TMOS
Symptoms:
Automatic ISO digital signature checking occurs but is not required when FIPS license active.
The system logs an error message upon an attempt to install or update the BIG-IP system:
failed (Signature file not found - /shared/images/BIGIP-13.1.0.0.0.1868.iso.sig)
Conditions:
When the FIPS license is active, digital signature checking of the ISO is automatically performed. This requires that both the ISO and the digital signature (.sig) file are uploaded to the system.
Impact:
Installation does not complete if the .sig file is not present or not valid. Installation failure.
Workaround:
To validate the ISO on the BIG-IP system, follow the procedure described in K24341140: Verifying BIG-IP software images using .sig and .pem files :: https://support.f5.com/csp/article/K24341140.
Component: TMOS
Symptoms:
Changing existing self IP addresses in bigip_base.conf file directly. After uploading the changed configuration file, BIG-IP routing service provides out of date Self IP route information to dependent services.
Conditions:
- Self IP address is configured on the BIG-IP system.
- Manually change the IP address of a self IP in bigip_base.conf file.
- Load changed configuration via tmsh.
Impact:
Different services have different route information:
-- tmsh table - has the old route.
-- Dynamic routing - hHas the old and new routes.
-- Kernel table - has the new route.
Workaround:
There are two workarounds, preventive and corrective.
Preventive:
Do not manually change self IP addresses in bigip_base.conf file. It is not recommended way to add/change BIG-IP configuration. Use GUI or tmsh instead.
Corrective:
If changed configuration is uploaded. In GUI or tmsh, delete changed self IP address, and then create a self IP address with old IP address and delete it as well. Now, all affected routes are removed.
Component: Access Policy Manager
Symptoms:
Mcpd crashes after a config sync.
Conditions:
This can occur during configuration validation when APM is configured.
Impact:
Mcpd may take too long to validate the APM configuration and is killed by watchdog, causing a core
Workaround:
Use the Visual Policy Editor to configure access policy instead of tmsh commands.
The Visual Policy Editor does not allow policies to be created if they contain loops.
Component: Access Policy Manager
Symptoms:
When SAML metadata file that contains certain UTF-8 characters other than ASCII is imported, SAML SAX Parser returns error
Conditions:
When SAML metadata file contains certain UTF-8 characters other than the ASCII set,
Impact:
SAML metadata file is not imported, and the system reports an error. SAML configuration on BIG-IP systems is impacted.
Workaround:
Remove the non-ASCII UTF-8 characters, and try the import operation again.
Component: Local Traffic Manager
Symptoms:
Payload data is collected at the HTTP_REQUEST event and finishes collecting (HTTP::release) when the NAME_RESOLVED event occurs. On releasing, data is forwarded to the serverside, triggering the HTTP_REQUEST_SEND event.
When trying to raise HTTP_REQUEST_SEND, the iRule queues it and returns IN_PROGRESS, because the system is already in the process of running TCLRULE_NAME_RESOLVED. (Nested iRules: TCLRULE_NAME_RESOLVED -> TCLRULE_HTTP_REQUEST_SEND)
Due to the IN_PROGRESS status, tcp_proxy skips forwarding HUDCTL_REQUEST to the serverside, but not the subsequent payload. So the HTTP filter considers this an invalid action.
Conditions:
-- Standard virtual server with iRules attached (for example, using the following configuration for a virtual server):
when HTTP_REQUEST {
HTTP::collect
NAME::lookup @10.0.66.222 'f5.com'
when NAME_RESOLVED {
HTTP::release
when HTTP_REQUEST_SEND {
log local0. "Entering HTTP_REQUEST_SEND"
-- Client sends two HTTP Post requests.
-- After the first request, the second connection is kept alive (for example, by using HTTP header Connection) so that the second request can reuse the same connection.
Impact:
The second request gets reset, and the system logs errors in the LTM log file.
Workaround:
To avoid nested iRules in this instance, simply remove the HTTP_REQUEST_SEND from the iRule.
Component: Local Traffic Manager
Symptoms:
The BIG-IP system maintains an SSL session cache for SSL (https) monitors. After changing the properties of an SSL monitor that might affect the operation of SSL, the BIG-IP continues to reuse an existing SSL session ID.
Conditions:
-- The BIG-IP system has cached session ID from previous SSL session.
-- SSL properties of monitor that might affect the operation of SSL are changed.
-- Monitor is using bigd.
Impact:
Sessions still use cached session ID. If session continues to succeed, session uses cached session ID till expiry.
Workaround:
-- Restart bigd.
-- Remove the monitor from the object and re-apply.
-- Use in-tmm monitors.
Component: TMOS
Symptoms:
With the administrator role, you have an option in TMUI to disable or restrict terminal access. If you disable or restrict access, the corresponding REST endpoint is neither disabled nor restricted.
Conditions:
Use TMUI as the admin, or as a user with the administrator role, and either of the following:
-- Disable terminal access.
-- Restrict access to TMSH.
Impact:
Users with the Administrator role can obtain shell access via REST.
With terminal access disabled:
-- If you attempt to login using SSH, you will not be to do so.
-- If you make a POST request to the /mgmt/tm/util/bash endpoint with a body that includes a command to run, that command will be run.
With access to TMSH restricted:
-- A POST request to the /mgmt/tm/util/bash endpoint that includes a body with a command to run will be run.
Workaround:
None.
Component: TMOS
Symptoms:
There is a tmrouted crash when new peer device is added.
Conditions:
The conditions under which this occurs are unknown.
Impact:
Core produced. Tmrouted crashes repeatedly. Dynamic routing for all route domains is temporarily disrupted.
Workaround:
Have MCP force load as described in K13030: Forcing the mcpd process to reload the BIG-IP configuration (https://support.f5.com/csp/article/K13030).
Component: Global Traffic Manager (DNS)
Symptoms:
Rolling DNSSEC Keys may stop generating when a BIG-IP system is on standby for a length of time
Conditions:
BIG-IP system is on standby for a length of time, in general, longer than twelve hours.
Impact:
Rolling DNSSEC keys can stop regenerating.
Workaround:
None.
Component: TMOS
Symptoms:
A tmsh core occurs when multiple tmsh instances are spawned and terminated quickly
/var/log/kern.log:
info kernel: tmsh[19017]: segfault ...
system messages in /var/log/messages:
notice logger: Started writing core file: /var/core/-tmsh ...
/var/log/audit:
notice -tmsh[19010]: 01420002:5: AUDIT - pid=19010 ...
Conditions:
This issue occurs intermittently in the following scenario:
1. Open multiple instances of tmsh using the following command pattern:
run util bash
run util bash
run util bash
run util bash
2. Quickly terminate them using Ctrl-D or by closing terminal.
Impact:
The tmsh utility crashes and produces a core file in the /shared/core directory. The BIG-IP system remains operational.
Workaround:
Restart tmsh if the problem occurs.
To prevent the issue from occurring: Do not quickly terminate tmsh instances using Ctrl-D.
Component: Local Traffic Manager
Symptoms:
If a cipher rule is changed, and a cipher group that uses the rule is attached to an SSL profile, the change does not take effect until something else on the SSL profile changes.
Conditions:
-- A cipher group is used by an SSL profile.
-- One of its cipher rules changes.
Impact:
Unexpected behavior occurs because the cipher rule change does not take effect immediately.
Workaround:
After changing the cipher rule that's used by a cipher group, make a change to any SSL profile that uses the associated cipher group.
Component: Local Traffic Manager
Symptoms:
Loading correct configuration with 'tmsh load /sys config' fails. The error message appears similar to the following:
01070726:3: Virtual Address /test/0.0.0.0 in partition test cannot be referenced by Virtual Server /Common/test-internal in partition Common.
Unexpected Error: Loading configuration process failed.
Conditions:
Creating internal virtual server, when 0.0.0.0 address exists on another partition.
Impact:
Inability to load config, with created internal virtual server.
Workaround:
Create internal virtual server first; then create the 0.0.0.0 address on a different partition.
Component: Local Traffic Manager
Symptoms:
LTM policies do not work if number of matches for a particular condition exceeds 127.
Conditions:
LTM policy that has a condition with more than 127 matches.
Impact:
LTM policy does not match the expected condition.
Workaround:
There is no workaround at this time.
Component: Local Traffic Manager
Symptoms:
Currently, bigd only informs mcpd of the state of a node on a state change. If the pool member status happens to be incorrect, this can cause the following symptoms.
-- Pool member status may be incorrect for a long time
-- Traffic may be directed to a pool member that is actually down.
Conditions:
-- Monitor is attached to pool member and bigd does not inform the state change event for log time in some of the corner cases.
-- No periodic events from bigd to mcpd.
Impact:
-- False monitor status in UI/CLI.
-- Large number of RST connections as traffic is directed to a pool member that is actually DOWN
Component: Device Management
Symptoms:
When upgrading to version 13.1.0.x or later, users who previously had permissions to make calls to iControl REST lose the ability to make those calls.
Conditions:
-- Upgrading to version 13.1.0.x or later.
-- Using iControl REST.
Impact:
A previously privileged user can no longer query iControl REST. In addition, some remotely authenticated users may lose access to the Network Map and Analytics view after the upgrade.
Workaround:
You can repair the current users permissions with the following process:
1) Delete the state maintained by IControlRoleMigrationWorker and let it rerun by restarting restjavad process:
# restcurl -X DELETE "shared/storage?key=shared/authz/icontrol-role-migrator"
# bigstart restart restjavad.
2) Update shared/authz/roles/iControl_REST_API_User userReference list to add all affected users' accounts using PUT:
# restcurl shared/authz/roles/iControl_REST_API_User > role.json
# vim role.json and add { "link": "https://localhost/mgmt/shared/authz/users/[your-user-name]" } object to userReferences list
# curl -u admin:admin -X PUT -d@role.json http://localhost/mgmt/shared/authz/roles/iControl_REST_API_User
Now, when you create a new user, the permissions should start in a healthy state.
Component: TMOS
Symptoms:
TMM crashes.
Conditions:
SessionDB is accessed in a specific way that results in an invalid state.
Impact:
TMM crashes. Traffic disrupted while tmm restarts.
Workaround:
None.
Component: Global Traffic Manager (DNS)
Symptoms:
In certain circumstances, attaching a BIG-IP monitor type to a non-BIG-IP server with already defined virtual servers is allowed by the system when it should not be allowed.
Conditions:
Attempting to attach a BIG-IP monitor type to a non-BIG-IP server.
Impact:
The BIG-IP monitor can be added to a non-BIG-IP server without error. This causes a configuration load error, such as after a reboot, tmm restart, or tmsh load sys config, and results in an error message such as:
-- localhost emerg load_config_files: "/usr/bin/tmsh -n -g load sys config partitions all gtm-only" - failed. -- Loading schema version: 12.1.3 Loading schema version: 12.1.5.1 01071033:3: Server (/Common/generic_server_object) contains monitor (/Common/bigiptest) which is an invalid type. Unexpected Error: Loading configuration process failed.
Workaround:
None.
Component: Device Management
Symptoms:
Occasionally, the BIG-IP Admin Utility TMUI fails to function correctly and produces the following error:
Error getting auth token from login provider.
This occurs when the BIG-IP REST Daemon restjavad fails to start up properly.
Conditions:
This error most often occurs on the first or second boot after upgrade, and more often on Virtual Edition BIG-IP platforms running on oversubscribed or slow hypervisors.
Impact:
TMUI and any other BIG-IP system components that rely on REST Workers such as: OpenID Connect key rotation discovery, portions of the TMOS Web Configuration Utility, and Guided Configuration (AGC and WGC) fail to function properly.
Workaround:
Restarting the BIG-IP REST daemons restjavad and restnoded will usually correct the problem. To do so, connect to the SSH console and issue the following two commands:
bigstart restart restjavad
bigstart restart restnoded
Component: Advanced Firewall Manager
Symptoms:
tmm restarts.
Conditions:
-- AFM DoS single endpoint (sweep or flood) vector is configured.
-- The attack is ongoing.
-- The attack vector is being mitigated in hardware (HW).
-- The vector is manually disabled.
Impact:
tmm can crash and restart. Traffic disrupted while tmm restarts.
Workaround:
If you do not want to mitigate, set the mitigation_threshold to infinite.
Note: Do not disable the single endpoint vectors when an attack is ongoing and the vector is being mitigated in HW.
Component: Global Traffic Manager (DNS)
Symptoms:
Cannot use iControl REST to create topology records when whitespace exist in a STATE name.
Conditions:
STATE name contains a space (e.g., New Mexico).
Impact:
Unable to create a topology record using iControl REST.
Workaround:
Use TMSH with quotes or escaping to create topology records for a STATE with whitespace in the name.
Component: TMOS
Symptoms:
If you run the command 'tmsh load sys config current-partition' in a partition that includes a folder, the command may return an error.
Conditions:
This occurs in the following scenario:
-- Create a partition.
-- Create a folder in that partition.
-- In the newly-created partition.
-- Save the configuration with the command 'save sys conf'.
-- In the same partition, run the following command to load the configuration: 'tmsh load sys config current-partition'.
Impact:
The load configuration process fails with an error that the folder does not exist.
Workaround:
There is no workaround at this time.
Component: TMOS
Symptoms:
Malformed JSON files may be present on vCMP host.
Conditions:
All needed conditions are not yet defined.
- vCMP is provisioned.
- Guests are deployed.
- Software versions later than 11.6.0 for both guest/host may be affected.
Impact:
Some vCMP guests may not show up in the output of the command:
tmsh show vcmp health
In addition, there might be files present named using the following structure:
/var/run/vcmpd/<guestname>/json/sys-(ha-status|provision|software).json.bad.
There is no functional impact to the guests or to the host, other than these lost tables, which are provided as a convenience to the vCMP host administrator.
Workaround:
None.
Component: Access Policy Manager
Symptoms:
The Network Access client caches the response for /pre/config.php.
Conditions:
-- APM is provisioned.
-- Network Access is configured.
Impact:
Caching the response for /pre/config.php might reveal configuration information. However, a URL is public information by definition. The only sensitive information revealed are server names, which have to be revealed in order for the client to know where to connect.
Workaround:
None.
Component: TMOS
Symptoms:
A vCMP guest usually provides the vCMP host with some guest health statistics as a convenience to the vCMP host administrator. These stats are:
-- mgmt/tm/sys/ha-status
-- mgmt/tm/sys/software/status
-- mgmt/tm/sys/software/provision
These tables are created by the host when host vcmpd queries the guest over the vmchannel using REST.
These RESTful queries may sometimes fail, causing the queried vCMP guest to be omitted in the display of the output of the following command: $ tmsh show vcmp guest
Conditions:
-- vCMP provisioned.
-- Guests are deployed.
-- Host vcmpd queries the guest over the vmchannel using REST.
Impact:
There is no functional impact to the guests or to the host, other than these lost tables.
-- Some vCMP guests may not show up in the output of the following command: tmsh show vcmp health
-- Some guests may appear with the wrong status in the GUI. Such as being grey when it should be green.
-- Files containing guest information, kept in:
/var/run/vcmpd/<guestname>/json/(sys-ha-status.json|sys-provision.json|sys-software.json) may be missing from that directory.
-- There might be files present there named using the following structure:
/var/run/vcmpd/<guestname>/json/sys-(ha-status|provision|software).json.bad.
Workaround:
There is no workaround at this time.
Component: Local Traffic Manager
Symptoms:
DHCP traffic can traverse between route domains, e.g., when working with a route domain with a parent. Under certain circumstances, this is not desired.
Conditions:
DHCP relay in use on a route domain with a parent relationship or strict isolation disabled.
Impact:
The DHCP server side flow might get established to the parent route domain, and will persist even after the route in its own route domain becomes available again.
Workaround:
There is no workaround at this time.
Component: Global Traffic Manager (DNS)
Symptoms:
TMUI (the GUI) is unable to change the Alias Address of DNS/GTM Monitors.
Conditions:
-- Using the GUI.
-- DNS/GTM monitors with alias address.
-- Attempting to change the Alias Address.
Impact:
Cannot change the Alias Address.
Workaround:
Use tmsh.
Component: Access Policy Manager
Symptoms:
When the BIG-IP system configured as OAuth AS has a missing OAuth Profile in the Access profile, the error log is not clear. It shows an error indicating 'OAuth mode is not set' instead of showing 'OAuth Profile is not configured'. In this case, the error 'OAuth mode is not set' means that the OAuth Profile is not associated in the Access profile of the virtual server acting as BIG-IP OAuth Authentication server.
Conditions:
-- The BIG-IP system is configured as OAuth AS.
-- The Access profile is not configured with OAuth profile.
Impact:
Confusing error message that leads to delay in troubleshooting the OAuth configuration.
Workaround:
Configure an OAuth Profile in the Access profile of the virtual server acting as BIG-IP OAuth AS.
Component: Local Traffic Manager
Symptoms:
The SSL filter will not always add the ALPN extension.
Conditions:
If the negotiated cipher is not HTTP/2 compliant, the SSL filter may not add the ALPN extension.
Impact:
The failure to add the ALPN extension may result in the failure to negotiate the proper protocol.
Workaround:
There is no workaround at this time.
Component: Application Security Manager
Symptoms:
There is an option to add 'secure' attribute to ASM cookies.
There are some specific cookies which this option does not apply on.
Conditions:
-- ASM policy is attached to the virtual server.
-- Internal parameter 'cookie_secure_attr' flag is enabled, along with either of the following:
+ Using HTTPS traffic.
+ The 'assume https' internal parameter is also enabled.
-- Along with one of the following:
+ Web Scraping' feature is enabled.
+ 'Bot Detection' feature is enabled.
+ The 'brute force' feature is enabled using CATPCHA.
Impact:
Some cookies do not have the 'secure' attributes.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
The default value for connection.syncookies.threshold should be set to 64000. Instead, this value defaults to 16384.
Conditions:
This issue may be encountered when a virtual server uses syncookies.
Impact:
The connection.syncookies.threshold value will be lower than intended, possibly resulting in lower performance.
Workaround:
Use tmsh to manually set the threshold value:
# tmsh modify sys db connection.syncookies.threshold value 64000
Component: Local Traffic Manager
Symptoms:
HTTP/2 connection in gateway scenario is reset when HTTP/2 preface makes it to the server instead of being consumed by the BIG-IP system. The backend connection to the server in gateway scenario is an HTTP/1 connection. The connection is reset when HTTP/2 preface is sent on the backend connection instead of being consumed by the BIG-IP system.
Conditions:
HTTP/2 gateway is configured.
Impact:
HTTP/2 connection reset is seen.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
BIG-IP systems provide a client-side HTTP/2 Gateway protocol implementation in conjunction with HTTP 1.x on a server side. A response can be cached on the BIG-IP system with a web acceleration profile. Sometimes a response served from cache is prematurely marked with END_STREAM flag, causing the client to ignore the rest of the response body.
Conditions:
BIG-IP system has a virtual server for which HTTP/2 and Web Acceleration profiles are configured.
Impact:
Some clients' browsers do not retry a resource, causing incorrect rendering of an HTML page.
Workaround:
Adding the following iRule causes the body to be displayed:
when HTTP_RESPONSE_RELEASE {
set con_len [string trim [HTTP::header value Content-Length]]
HTTP::header remove Content-Length
HTTP::header insert Content-Length "$con_len"
Component: TMOS
Symptoms:
Trying to create a SNMP trap may fail in the GUI with the following error message: An error has occurred while trying to process your request.
Conditions:
-- Trap destinations are configured using the GUI: When trap destinations are configured in the GUI, the trap name is generated using the destination IP address.
-- Traps of the same destination address were previously created and deleted.
Impact:
GUI parameter checking does not work as expected. BIG-IP Administrator is unable to create a SNMP trap session.
Workaround:
To work around this issue when using the GUI, remove all traps that have the same destination address as the new one that failed. Then re-add your destination.
Tip: You can use tmsh to create/delete/modify SNMP traps, which enables viewing of the generated names, making it easier to understand what error has occurred.
Component: TMOS
Symptoms:
Audit log messages sent from the primary blade to the secondaries are logged.
Conditions:
Multi-blade platform.
Impact:
Unnecessary messages in the log file.
Workaround:
None.
Component: TMOS
Symptoms:
Beginning in BIG-IP 12.0.0, the tmrouted process pushes dynamic routes directly to the Traffic Management Microkernel (TMM). This message indicates the system is shutting down and is expected behavior during the reboot or shutdown process. The appearance of this message on a stable running system may indicate an issue with tmrouted functionality.
System posts the following message in /var/log/ltm: 01010054:3: tmrouted connection closed
Conditions:
This message occurs when all of the following conditions are met:
-- You have configured the BIG-IP system to use dynamic routing.
-- The BIG-IP system is in the process of shutting down or rebooting.
Impact:
This message is benign, unless you view the message on a stable running system. In this case, the message may indicate an issue with the tmrouted process.
Workaround:
None.
Component: TMOS
Symptoms:
When using remote role groups to set user/role/partition information, user login fails, but logs in /var/log/secure indicate that authentication was successful.
Errors similar to the following appear in /var/log/ltm:
-- User restriction error: The administrator, resource administrator, auditor and web application security administrator roles may not be restricted to a single partition.
-- Input error: invalid remote user credentials, partition does not exist, broken-partition
Errors similar to the following appear in /var/log/secure:
tac_authen_pap_read: invalid reply content, incorrect key?
Conditions:
Using remote role groups to set user/role/partition information for remote users, and either of the following:
-- A remote user is configured with the role of administrator, resource administrator, auditor, or web application security administrator, with access to a particular partition, rather than all. (These roles require access to all partitions.)
-- A remote user is configured with partition access set to a partition that does not exist on the BIG-IP system.
Impact:
The messages in /var/log/secure may be confusing and make it more difficult to diagnose the login failure.
Workaround:
Check /var/log/ltm for more specific error messages.
Component: TMOS
Symptoms:
Unable to set VET, AZOT, and AZOST timezones
Conditions:
This occurs under normal operation.
Impact:
Cannot set these timezones.
Workaround:
Use the following zones with the same offset:
The AZOT timezone is the same offset as
N – November Time Zone.
The AZOST timezone is the same offset as
Z – Zulu Time Zone,
GMT – Greenwich Mean Time,
WET – Western European Time.
The VET timezone is the same offset as
AST – Atlantic Standard Time,
CDT – Cuba Daylight Time, CLT – Chile Standard Time,
EDT – Eastern Daylight Time,
FKT – Falkland Island Time,
Q – Quebec Time Zone.
Component: Advanced Firewall Manager
Symptoms:
When a rule is selected and the 'Reset Count' button is clicked, the command is executed but rule stats are not updated in the GUI.
Conditions:
This occurs when resetting the rule hit count stats in the GUI.
Impact:
Incorrect (stale) statistics are seen.
Workaround:
Refresh the page.
Component: TMOS
Symptoms:
The following sections of the GUI can report incorrect or higher than expected CPU utilization:
- The "download history" option found in the Flash dashboard
-- Statistics :: Performance :: Traffic Report (section introduced in version 12.1.0).
Values such as 33%, 66%, and 99% may appear in these sections despite the system being completely idle.
Conditions:
HT-Split is enabled (default for platforms that support it).
Impact:
The CPU history in the exported comma-separated values (CSV) file does not match actual CPU usage.
Workaround:
You can obtain CPU history through various other means. One way is to use the sar utility.
In 12.x and higher versions:
sar -f /var/log/sa6/sa
or for older data
sar -f /var/log/sa6/sa.1
The oldest data is found compressed in /var/log/sa6 and must be gunzipped before use.
In 11.x:
sar -f /var/log/sa/sa
or for older data
sar -f /var/log/sa/sa.1
The oldest data is found compressed in /var/log/sa and must be gunzipped before use.
Live CPU utilization can also be obtained through the Performance Graphs, SNMP polling, iControl polling, various command-line utilities such as top, etc.
Component: TMOS
Symptoms:
When IPsec tunnels to several different peers are configured using a single ipsec-policy in interface mode, the tunnels will be unreliable or may not start.
Conditions:
Several traffic-selectors that are associated with different tunnels reference the same interface mode IPsec policy.
Note: It is not possible to create this configuration when the IPsec policy is tunnel or transport mode.
Impact:
IPsec tunnels may start but fail after a period of time. In other cases, IPsec tunnels may not start at all.
Workaround:
(1) Create a unique ipsec-policy configuration object for each remote peer and traffic-selector.
(2) Use tunnel mode. It is not possible to create this configuration when the IPsec policy is tunnel or transport mode.
Component: TMOS
Symptoms:
An idle-timeout occurs while running a sub-process in interactive mode, resulting in a log message. tmsh logs a benign but ominous-looking critical error to the console and to /var/log/ltm if a tmsh command reaches idle timeout and a spawned sub-process is still running.
The errors in /var/log/ltm begin with the following text:
'boost assertion failed'
Conditions:
-- tmsh command reaches idle timeout.
-- Spawned sub-process is still running.
Impact:
Although the wording indicates a failure, the message is benign and you can safely ignore it.
Workaround:
None.
Component: TMOS
Symptoms:
Mcpd crashes.
Conditions:
The file-store path is missing with specific configuration file which is needed by mcpd while booting.
Impact:
Traffic and control plane disrupted while mcpd restarts.
Component: Advanced Firewall Manager
Symptoms:
On a vCMP platform with host and guest using different BIG-IP versions, when DNS is under load greater than the AFM-configured rate limit, certain IPv4 packets are categorized as IPv6 atomic fragments and may be dropped due to rate limits.
Conditions:
-- vCMP platform with host and guest using different BIG-IP versions.
-- AFM enabled.
-- DNS load greater than AFM-configured rate limit for IPv6 atomic fragments (default 10 KB).
Impact:
May result in lower than expected DNS load test results.
Workaround:
You can use any of the following workarounds:
-- Disable AFM.
-- Increase detection limit for IPv6 atomic fragments under AFM.
-- Disable hardware offload with sys db Dos.VcmpHWdos.
Note: For AFM HW DoS protection, the host and vCMP guest must be the same version, disable hardware DoS checking on the vCMP guest to prevent this issue. To do so, set sys db dos.forceswdos to 'true'.
Component: TMOS
Symptoms:
An error message appears in the GUI reading 'This device is not operational because the loaded configuration contained errors or unlicensed objects. Please adjust the configuration and/or the license, and re-license the device.' Examination of the configuration and license shows that there are no configuration error or unlicensed configuration objects. The device is operational.
Conditions:
The BIG-IP system is licensed and the configuration loaded.
Impact:
Error message appears in the GUI stating that the device is not operational. However, the device is operational.
Workaround:
Restart mcpd by running the following command:
bigstart restart mcpd
Component: TMOS
Symptoms:
During platform migration from a physical BIG-IP system to a BIG-IP vCMP guest, the load fails with one of these messages:
01070687:3: Link Aggregation Control Protocol (LACP) is not supported on this platform. Unexpected Error: Loading configuration process failed.
01070338:3: Cannot create trunk [name of trunk], maximum limit reached Unexpected Error: Loading configuration process failed.
Conditions:
-- The source device is a physical BIG-IP device with one or more trunks with or without LACP in its configuration.
-- The destination device is a vCMP guest.
Impact:
The platform migration fails and the configuration does not load.
Workaround:
You can use either of the following Workarounds:
-- Remove all trunks from the source configuration prior to generation of the UCS.
-- After the UCS load fails, edit the configuration manually on the destination to remove trunk references, and then reload the configuration.
Component: TMOS
Symptoms:
When you load a UCS file using the platform-migrate parameter, the mgmt-dhcp value (enabled, disabled, or unset) will overwrite the value on the destination. Depending on the effect, this could change the destination's management IP and default management route.
If the UCS does not have mgmt-dhcp explicitly written out, note that its value is treated as the default for the local system, which varies by the type of system. On Virtual Edition (VE) platforms, the default is to enable DHCP. On all other platforms, the default is to disable DHCP.
Conditions:
This occurs when loading a UCS using the platform-migrate parameter:
tmsh load sys ucs <ucs_file_from_another_system> platform-migrate
Impact:
Changing the mgmt-dhcp value on the destination can result in management changing from statically configured to DHCP or DHCP to statically configured. This can result in loss of management access to the device, requiring in-band or console access.
Workaround:
If you want to reset the target device to use a static IP, run the following commands after loading the UCS with the platform-migrate command:
tmsh modify sys global-settings mgmt-dhcp disabled
tmsh create sys management-ip <ip>/<mask>
tmsh delete sys management-route default
tmsh create sys management-route default gateway <ip>
Component: TMOS
Symptoms:
The BIG-IP system reuses ephemeral ports quickly, which might cause connections to fail, because those ports are in TIME-WAIT state on the server.
Conditions:
-- FTP active mode is used.
-- Virtual server source-port change.
-- Running on an affected platform:
--- BIG-IP 5000 series (C109)
--- BIG-IP 7000 series (D110)
--- BIG-IP 10000s/10050s/10200v/10250v (D113, D112)
--- B4300 / B4340N VIPRION blades (A108, A110)
Impact:
FTP active mode connections might fail.
Workaround:
There is no complete workaround. However, you can mitigate the issue using the default setting source-port=preserve on the virtual server.
As alternatives, you can also try the following:
-- Switching client traffic to Passive FTP.
-- Enabling TIME_WAIT reuse on the FTP server, if the operating system supports it.
Component: Application Visibility and Reporting
Symptoms:
Using the TMSH for AVR reports can fail if it contains partition based entities, even with an administrator user (which should have permissions to all partitions).
Conditions:
Using the TMSH for querying partitioned based stats with an administrator user.
Impact:
AVR reports via TMSH will fail when using partition based entities.
Workaround:
None.
Component: TMOS
Symptoms:
When you access the vCMP guest instance after you deploy the system, the instance uses the hostname bigip1.
Conditions:
This issue occurs when all of the following conditions are met:
-- The BIG-IP system is running v11.6.0 or earlier.
-- You configure a vCMP guest instance that is running BIG-IP v11.6.0 or later.
-- You have configured the vCMP guest instance with a hostname other than bigip1.
-- You deploy the vCMP guest instance.
Impact:
The vCMP guest does not use the configured hostname.
Workaround:
-- In tmsh, run the following commands, in sequence:
mv cm device bigip1 HOSTNAME
save sys config
-- Rename the device name in the GUI.
Component: Local Traffic Manager
Symptoms:
DHCP statistics are incorrect if DHCP server is down while a new virtual server is created. And when the DHCP server comes back up, the current pending transactions are incorrect.
Conditions:
-- DHCP relay configured.
-- DHCP server is down while the virtual server is created.
Impact:
The 'current pending transactions' for the DHCP server is incorrect if the DHCP server is down while a new virtual server is created.
Workaround:
None.
Component: TMOS
Symptoms:
MCPD successfully loads configuration when it's not able to decrypt DNSSEC Key generation.
Conditions:
MCPD loads configuration with DNSSEC Key generation encrypted by master key which has been changed.
Impact:
The configuration successfully loads but BIG-IP is not able to sign Resource Records.
Component: TMOS
Symptoms:
The files
.bash_logout
.bash_profile
.bashrc
in a user's home directory are overwritten when that user's password is changed.
Conditions:
Change a user's password.
Impact:
Customizations to these files would be lost on password change. This only applies to users with advanced shell access.
Workaround:
Back up the files to a different location before making a password change.
Component: TMOS
Symptoms:
When running gtm_add from one BIG-IP system to another, if the system being added already has the same DNSSEC key (dictated by DNSSEC key name), and you synchronize the FIPS card, then the FIPS card is wiped out (as expected), but the key is not re-added.
Conditions:
-- There is an existing DNSSEC key on one system.
-- A second system has a DNSSEC key of the same name.
-- Run gtm_add, with instructions to synchronize FIPS cards.
Impact:
No DNSSEC key of that name is present on FIPS card.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
TMM cores on SIGSEGV during normal operation.
Although the exact triggering conditions are unknown, it might that when a connection is aborted in a client-side iRule, the reported log signature may indicate its occurrence:
tmm3[11663]: 01220009:6: Pending rule <irule_name> <HTTP_REQUEST> aborted for <ip> -> <ip>.
Conditions:
Specific conditions that trigger this issue are unknown.
Impact:
Traffic disrupted while tmm restarts.
Workaround:
None.
Component: TMOS
Symptoms:
Overwrite flag in key/certificate management iControl functions is ignored and might result in errors.
Conditions:
If there is an existing key/certificate, and the key/certificate management iControl/SOAP functions are used to overwrite the key/certificate by setting the overwrite flag, the flag is ignored, and an error is returned.
Impact:
Key/certificate overwrite using iControl operations might fail.
Component: TMOS
Symptoms:
IPv6 virtual servers may not be reachable on specific Virtual Edition (VE) instances, based on the interfaces used by that instance.
Conditions:
VE with IPv6 virtual servers configured. The VE instance must have SR-IOV interfaces, or KVM virtio interfaces backed by macvtap.
Impact:
Special configuration procedure is required.
Workaround:
Follow the defined configuration procedure by explicitly listing the VLANs for which this VIP is enabled. For example, in TMSH:
'modify ltm virtual <name> vlans-enabled'
'modify ltm virtual <name> vlans add { <vlans> }'
Component: TMOS
Symptoms:
In the TMUI on any iRule editing page, brace matching within a switch statement can fail if a literal string is surrounded with braces.
Conditions:
Use a literal string surrounded with curly braces for a case/pattern within a switch statement.
Impact:
Incorrect brace matching.
Workaround:
Instead of surrounding the literal string with braces, use double quotes.
Component: Access Policy Manager
Symptoms:
Internally detected issues can trigger a series of error log messages. The logs look alarming, but can be considered diagnostic in the case there is an actual behavioral issue that needs to be analyzed.
The system reports the following messages in /var/log/apm:
-- err tmm1[11197]: 01490514:3: 00000000: Access encountered error: ERR_VAL. File: ../modules/hudfilter/access/access.c, Function: access_slowpath_security_check, Line: 6648
-- err tmm8[18022]: 01490514:3: 00000000: Access encountered error: ERR_ARG. File: ../modules/hudfilter/access/access.c, Function: access_sanitize_uri, Line: 16406
-- err tmm8[18022]: 01490514:3: 00000000: Access encountered error: ERR_ARG. File: ../modules/hudfilter/access/access.c, Function: access_check_uri_type, Line: 11219
-- err tmm8[18022]: 01490514:3: 00000000: Access encountered error: ERR_ARG. File: ../modules/hudfilter/access/access.c, Function: access_process_state_client_classify_req, Line: 3308
-- err tmm8[18022]: 01490514:3: 00000000: Access encountered error: ERR_ARG. File: ../modules/hudfilter/access/access.c, Function: hud_access_handler, Line: 2311
Conditions:
An internal software API call triggers an unexpected result.
Impact:
Logs might give the appearance of many issues, even if there are no behavioral anomalies.
Workaround:
None.
Component: TMOS
Symptoms:
On the affected slot, the vCMP guest is unable to pass traffic to or from the VLANs. If the guest has multiple slots, the CMP state logged in /var/log/tmm on that slot differs from the CMP state logged by other slots of the same guest.
In the vCMP guest, 'tmsh show net interface -hidden' shows 0.x interfaces for the affected slot that differ from the 0.x interfaces shown by 'tmsh show vcmp guest all-properties' on the vCMP hypervisor for the same guest slot.
Conditions:
The MCPD daemon on one of the blades of the vCMP hypervisor crashes or restarts.
Impact:
The vCMP guests that are still running since before the MCPD daemon restarted may be unable to communicate to VLAN networks. Incoming traffic may also be affected, even though the vCMP guest has other functional slots to process traffic.
Workaround:
On the hypervisor, modify the vCMP guest configuration to not run on the affected slot. Wait to confirm the vCMP guest has stopped on the affected slot. Then modify the vCMP guest to run on the previously affected slot.
Alternatively, modify the vCMP guest to the Configured state, and wait to confirm the vCMP guest has stopped on all slots. Then return the vCMP guest to the Deployed state.
Component: Application Security Manager
Symptoms:
REmote logs are missing. The following log messages appears in bd.log and asm.log:
ASM configuration error: event code L3350 Failed to write to remote logger vs_name_crc 1119927693 LoggingAccount.cpp:3348`remote log write FAILED res = -3 <Failed to send remote message (remote server not responding)> errno <Message too long>.
Conditions:
-- A remote logger configured for UDP.
-- Max message length of 64 KB.
Impact:
Missing logs in the remote logger.
Workaround:
You can use either of the following workarounds:
-- Change the remote logger to TCP.
-- Reduce the message length to 1 KB.
Component: Local Traffic Manager
Symptoms:
Forwarding flow doesn’t expire and leaks a connflow object.
Conditions:
Conditions to hit this are a route change on forwarded flows.
Impact:
Memory leak.
Workaround:
Component: TMOS
Symptoms:
When the mcpd configuration database is not available password policy is not enforced when changing passwords for the user 'root' using the command-line utility 'passwd' utility.
Conditions:
-- Advanced shell access
-- mcpd is not available.
-- Change root password with the 'passwd' utility.
Impact:
Root password may be set to a string that does not comply with the current password policy.
Workaround:
None.
Component: TMOS
Symptoms:
You encounter connectivity issues to management interface.
Conditions:
MTU on the intermediate route is less than the management interface's MTU and the response packets have the DF flag set.
Impact:
Connectivity issues to management interface.
Workaround:
None.
Note: Although there is no workaround for this module, you can disable auto last hop and configure a default gateway to avoid this issue.
For more information see K52592992: Overview of the Auto Last Hop feature on the management interface, available at
https://support.f5.com/csp/article/K52592992.
Component: Access Policy Manager
Symptoms:
OneConnect is a feature that reuses server-side connections. When WEBSSO is enabled, it always creates a new server-side connection, and doesn't reuse pooled connections.
Conditions:
WEBSSO and OneConnect.
Impact:
Idle serverside connections that should be eligible for reuse by the virtual server are not used. This might lead to build-up of idle serverside connections, and may result in unexpected 'Inet port exhaustion' errors.
Workaround:
None.
Component: TMOS
Symptoms:
If an iControl REST asynchronous task is running, other iControl REST queries (synchronous or asynchronous) will wait until the asynchronous task completes before executing. If the asynchronous task is long-running, subsequent requests will block for a long time.
Conditions:
-- Executing an iControl REST task asynchronously.
-- Performing further iControl REST tasks (synchronous or asynchronous) while the asynchronous task is still running.
Impact:
Potential (and unexpected) long wait times while running a task asynchronously.
Workaround:
None.
Component: TMOS
Symptoms:
Config sync after sub-partition config changes results extra lines in the partition's conf file.
Conditions:
Make changes under any partition except /Common and then config sync without overwrite.
Impact:
/config/partitions/partition_name/bigip_base.conf in the partitions folder has trunk and ha-group configuration. /config/bigip_base.conf no longer has the trunk and ha-group configuration.
Workaround:
'Sync Device to Group' with 'Overwrite Configuration' enabled.
Component: Local Traffic Manager
Symptoms:
Connection setup when using FastL4 profile and connection mirroring takes longer than previous versions.
Conditions:
FastL4 profile with connection mirroring.
Impact:
Slight delay during connection setup.
Workaround:
Disable tm.fastl4_ack_mirror. Optionally, enable tm.fastl4_mirroring_taciturn for signal to noise ratio improvements. This helps resolve connection setup latency.
Component: TMOS
Symptoms:
Under some conditions, the BIG-IP system might fail to report statistics over time. This can manifest as statistics reporting unchanging statistics (e.g., all zeroes (0)), or as sudden spikes in traffic, or as negative deltas in some counters.
The system performance graphs will also appear to have gaps / be missing data at the times that this occurs.
Conditions:
This occurs when there are frequent changes occurring to the underlying statistics data structures. This might occur under the following conditions:
-- The system is spawning/reaping processes on a frequent basis (e.g., when there is a large number of external monitors).
-- iRules are frequently using 'SSL::profile' to select different SSL profiles on a virtual server (this can cause per-virtual server, per-profile statistics to be created and deleted on a regular basis).
Impact:
Statistics fail to merge, which results in incorrect view of system behavior and operation.
Workaround:
This issue has two workarounds:
1. Reduce the frequency of changes in the statistics data structures. The specific action to take depends on what is triggering them. To do so, use any or all of the following:
-- Reduce the frequency of configuration changes.
-- Reduce the use of 'SSL::profile' in iRules.
-- Reduce the number/frequency of processes being spawned by the system.
2. Switch statistics roll-ups to the 'slow_merge' method, which causes the system to spend more CPU merging statistics. To do so, set the 'merged.method' DB key to 'slow_merge' using the following command:
tmsh modify sys db merged.method value slow_merge.
Component: WebAccelerator
Symptoms:
When tmm allocates memory, it adds up stats per memory type allocated. AAM is not properly marking memory type for strings objects, affecting other types of memory stats depending on configuration and release.
Conditions:
AAM is provisioned and there are virtuals in BIG-IP configuration which have web acceleration profiles associated with one or more AAM policies.
Impact:
Stats for some types of memory can be skewed causing troubleshooting issues.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
Nitrox crypto hardware may attempt soft reset to clear a stuck condition while already engaged in a soft reset attempt.
Conditions:
Soft reset is needed to clear a stuck condition occurring in the timeframe during which another soft reset is occurring.
Impact:
The initial soft reset attempt does not complete as the process is restarted by the new attempt.
Workaround:
Correct the condition resulting in the need for the soft reset to clear the stuck condition or disable hardware-based crypto acceleration by setting db variable 'tmm.ssl.cn.shunt' to disable.
To disable hardware-based crypto acceleration issue the following command:
tmsh modify sys db tmm.ssl.cn.shunt value disable
Note: Disabling hardware-based crypto acceleration results in all crypto actions being processed in software, which might result in higher CPU and memory usage based on traffic patterns.
Component: TMOS
Symptoms:
When there is a large number annotations in /var/annotate, tomcat might run out of memory when the dashboard requests the annotations.
The tomcat process logs an out-of-memory error, and the dashboard reports an error:
-- SEVERE: Servlet.service() for servlet org.apache.jsp.dashboard.annotations_jsp threw exception
java.lang.OutOfMemoryError: Java heap space.
-- Unrecoverable Communications Error. Please close this window and log in via the BIG-IP Configuration Utility.
Conditions:
-- Large number of configuration events, i.e., too many annotations in /var/annotate.
-- Click anything other than the default '5 minutes' tab.
Impact:
The the dashboard attempts to load all annotations into memory, but cannot. Dashboard must be restarted and can only be used at the 5-minute zoom level.
Workaround:
-- Delete the files in /var/annotate.
-- Increase the tomcat memory:
provision.tomcat.extramb = 320
Component: Access Policy Manager
Symptoms:
If APM is configured to protect multiple VMware resources (VCS servers), you can launch desktops and applications only from the first resource. Attempts to launch desktop or applications from other resources result in error.
Conditions:
-- APM is configured to protect multiple VMware resources (VCS servers).
-- You attempt to launch a desktop or application using the native VMware client.
Impact:
Cannot access desktops and applications from multiple VMware back-ends.
Workaround:
Use HTML5 client instead.
Component: Access Policy Manager
Symptoms:
IPv6 clients connecting to APM virtual server that renders some page, e.g., logon page, webtop, or message box, might get connection resets.
Conditions:
IPv6 client has the last 4 bytes of the IP address set to some special-purpose address, e.g., multicast address.
Impact:
Client connection is reset.
Workaround:
Change the last 4 bytes of the client IPv6 address to avoid the IPv4 special-address range.
Component: TMOS
Symptoms:
Evaluation features cause perpetual features to expire when the evaluation license expires.
Conditions:
-- Perpetual license with an evaluation/demonstration add-on feature.
-- The add-on license expires or is expired.
Impact:
When an evaluation/demonstration add-on license expires, features included in both the evaluation add-on as well as the regular, perpetual license stop working.
This behavior is covered in F5 article K4679: BIG-IP evaluation and demonstration licenses do expire :: https://support.f5.com/csp/article/K4679.
Workaround:
To work around this issue, activate the license from the command line:
When reactivating an existing license, and deactivating an expired evaluation license key, specify the base registration key and add-on (if any), and use the -i option for the expired evaluation license key in the get_dossier command.
For example, if the expired evaluation license key is ABCDEFG-ZZZZZZZ, use the following command:
get_dossier -b ABCDE-ABCDE-ABCDE-ABCDE-ABCDEFG -a ABCDEFG-ABCDEFG -i ABCDEFG-ZZZZZZZ
You can find these steps detailed in K2595: Activating and installing a license file from the command line :: https://support.f5.com/csp/article/K2595. This part in particular is required to work around this issue
Component: Access Policy Manager
Symptoms:
If you start to create an Access Profile and you set the Profile Type to SSO, you cannot complete the configuration from the New Profile screen unless an SSO configuration already exists.
Conditions:
Attempt to create a new Access Profile of Type SSO without having an existing SSO to select.
Impact:
Cannot create a new Access Profile of Type SSO until an SSO Configuration has been created and then selected for use within Access Profile.
Workaround:
To work around this problem, you can create an SSO Configuration prior to creating the new Access Profile of Type SSO.
Component: TMOS
Symptoms:
In rare BigIP configuration scenarios, TMM may crash during its startup process when the tunnel configurations are loaded.
Conditions:
During TMM startup, a tunnel is created, then immediately removed during the configuration load period, when TMM neighbor messages may be in flight via the tunnel. When the race condition fits, the neighbor message may land on an invalid tunnel.
Impact:
TMM crash in rare race conditions.
Workaround:
None.
Component: TMOS
Symptoms:
TACACS remote auth fails when the TACACS secret contains the '#' character.
Conditions:
TACACS secret contains the '#' character.
Impact:
TACACS remote auth fails.
Workaround:
Do not use the '#' character in the TACACS secret.
Component: TMOS
Symptoms:
If users with the certain roles change their password, the BIG-IP system does not detect that it is out-of-sync with its peer and does not trigger an automatic sync:
Conditions:
-- Multiple BIG-IP devices in a Device Service Cluster that sync configurations with each other.
-- A user with one of the following roles logs in and changes their password:
+ guest
+ operator
+ application-editor
+ manager
+ certificate-manager
+ irule-manager
+ resource-admin
+ auditor
Impact:
The system does not detect that it is out of sync with its peer, and does not report this condition. If automatic sync is enabled, a sync does not automatically occur.
Workaround:
Force a full sync to the peer systems.
Component: TMOS
Symptoms:
The system does not require setting a pool's min-up-members greater than 0 (zero) when also using gateway-failsafe-device on the same pool.
Conditions:
A pool's min-up-members is 0 when gateway-failsafe-device is set.
Impact:
Failure to set min-up-members greater than 0 when using gateway-failsafe-device might cause errors. The tmm might crash.
Workaround:
Set min-up-members greater than 0 when using gateway-failsafe-device.
Component: TMOS
Symptoms:
cli global-settings { service number } appears to have no effect.
Conditions:
cli global-settings { service number } command.
Impact:
Has no effect.
Workaround:
None.
Component: Local Traffic Manager
Symptoms:
The BIG-IP system erroneously performs a SNAT translation after the SNAT translation address has been disabled. As a result of this issue, you may encounter the following symptom:
A network trace capturing the affected traffic on a BIG-IP system shows traffic continues to egress the BIG-IP system using the disabled SNAT translation address.
Conditions:
This issue occurs when the following condition is met: A SNAT translation address is configured, but disabled.
Impact:
Traffic egresses the BIG-IP system with the disabled SNAT translation address.
Workaround:
To work around this issue, you must delete the affected SNAT configuration instead of disabling it. To do so, perform the following procedure:
Impact of workaround: Deleting the affected SNAT configuration removes it entirely from the BIG-IP configuration. If you require the SNAT configuration later, you must recreate it manually.
BIG-IP 11.x/12.x
1. Log in to the tmsh utility.
2. Delete the affected SNAT configuration by entering the following command: delete /ltm snat <affected SNAT name>.
For example, to delete the test-315765 SNAT configuration, you would enter the following command: delete /ltm snat test-315765.
3. Save the modified configuration by entering the following command:
save /sys config
BIG-IP 9.x through 10.x
1. Log in to the command line.
2. Delete the affected SNAT configuration by entering the following command: bigpipe snat <affected SNAT name> delete.
For example, to delete the test-315765 SNAT configuration, you would enter the following command: bigpipe snat test-315765 delete.
3. Save the modified configuration by entering the following command: bigpipe save all.
Component: Global Traffic Manager (DNS)
Symptoms:
The zrd process exits and cannot be restarted.
Conditions:
This occurs when the journal is out-of-sync with the zone.
Impact:
The zrd process cannot be restarted.
Workaround:
Before beginning, ensure that no one else is making config changes (i.e., consider making changes during a maintenance window).
I) On a working system, perform the following:
1. # rndc freeze $z
(Do this for all nonworking zones. Do not perform the thaw until you finish copying all needed files to the nonworking system.)
2. # tar zcvf /tmp/named.zone.files namedb/db.[nonworking zones].
3. # rndc thaw $z
II) On each nonworking system, perform the following:
1. # bigstart stop zrd; bigstart stop named
2. Copy the nonworking /tmp/named.zone.files from a working GTM system.
3. # bigstart start named; bigstart start zrd.
(Before continuing, review /var/log/daemon.log for named errors, and review /var/log/gtm for zrd errors0.)
Repeat part II until all previously nonworking systems are working.
III) On a working GTM system, run the following command:
# touch /var/named/config/named.conf.
一身肌肉的钥匙扣 · Python对文本进行分句_python 分句 1 月前 |
一身肌肉的钥匙扣 · Data governance beyond SDX: Adding third party assets to Apache Atlas - Cloudera Blog 9 月前 |
一身肌肉的钥匙扣 · javascript - In Redux-saga, how to get a result from a yield call to a Promise? - Stack Overflow 9 月前 |
一身肌肉的钥匙扣 · 如何计算 SASS 中的百分比减 px? 开发文档 11 月前 |
痴情的可乐 · C#实体框架:关键字不支持:‘端口’开发者社区 3 分钟前 |