This article focuses on troubleshooting delays in script execution when using a reverse proxy over HTTPS. We will explore the potential reasons for these delays and provide possible solutions to resolve the issues.
Understanding the Problem:
In older versions of ClearSCADA and Geo SCADA Expert, a statically compiled implementation of TLS was used for most connections, except for HTTPS connections created by ViewX. These connections, utilized by Mimic scripts running in ViewX to link to the server, relied on Microsoft libraries for HTTP/HTTPS communication. In contrast, newer versions of Geo SCADA Expert now use Microsoft TLS libraries (SCHANNEL) for all underlying connections. This transition enhances security by providing stronger encryption and regular security updates through the operating system. However, it also means that TLS connections must comply with the security configurations dictated by Security Policies, including default, local, and domain-defined settings.
Possible Reasons for Script Delays:
1- Certificate name mismatch.
The certificate name must match the server's name specified in the ViewX client configuration. If not, it will trigger a Windows check for certificate revocation, which is done if the names do not match.
First, launch IIS and verify that the "issued to" field of the self-signed certificate matches the server name:
2- The set-up delay is affecting ViewX client connection to the server.
Verify that in the Configure Connections dialog, the 'Connection Security | Validate Server Certificate' option is disabled. Additionally, ensure that when connecting to Geo SCADA Expert servers, the 'Use Temporary Certificates' option is enabled in the Server Configuration tool under 'System Configuration | Connection Security.'
If the server machines use self-signed certificates, please clear the 'Validate Server Certificate' check box. In this case, the 'Use Temporary Certificates' check box must also be selected in the Connection Security section of the Server Configuration Tool on the server machines.
For more details regarding the connection security settings, please refer to the manual available here.
3- The set-up delay is affecting server-server partner links.
Please verify that the 'Validate Partner Certificate' option is disabled in the Server Configuration under 'System Configuration | Partners | Connection Security' tab. If the other server machines are using self-signed certificates, kindly clear the check box. In this case, ensure that the 'Use Temporary Certificates' check box is selected in the Connection Security section (see previous picture) within the System Configuration branch of the Server Configuration Tool on those other server machines.
Please refer to the manual here for more information regarding the connection security tab.
4- Specific GPO Policies may causing the delays.
Please be aware that altering these policies may affect the connection validation of other applications and may not be suitable for those applications. Additionally, it can impact other client hosts within the domain. Please verify with experts before making any changes to the group policies.
Please note that this issue only occurs on systems that do not have internet access. On systems with internet access, the CRL lookups will succeed and be much faster than 15 seconds (default setting).
The CRL (Certificate Revocation List) lookups are used to check if a digital certificate has been revoked by the Certificate Authority (CA) before establishing a secure connection. When a client tries to connect to a server using an encrypted protocol (like SSL/TLS), it needs to ensure that the server's certificate is still valid and has not been revoked. when the system has internet access, it can quickly retrieve the CRL from certificate authority server. However, if the system lacks internet access, it cannot perform this lookup.
In the absence of internet access, the client may wait for a timeout period while trying to reach the CRL endpoint, leading to a significant delay before it either fails the connection or moves on to other checks.
The suggested changes that have proven effective in addressing the scripting delay are as follows:
Computer Configuration >> Policies >> Windows Settings >> Security Settings >> Public Key Policies >> Certificate Path Validation Settings >> Network Retrieval
Default URL retrieval timeout in seconds = 1
Default path validation cumulative timeout in seconds = 2
Allow issuer certificate retrieval during path validation = Disabled
Computer Configuration >> Policies >> Administrative Templates >> System >> Internet Communications Management >> Internet Communications Settings
Turn off Automatic Root Certificates Update = Enabled
It is possible to identify this issue from Windows events by enabling CAPI2 logging. You can view the Microsoft SCHANNEL path validation checks in the Event Viewer under 'Application and Services | Microsoft | Windows | CAPI2 | Operational' events.
To Enable Logging:
wevtutil.exe sl Microsoft-Windows-CAPI2\/Operational \/e:true
To Save the Log to a File:
wevtutil.exe epl Microsoft-Windows-CAPI2\/Operational filename.elf
To Disable Logging:
wevtutil.exe sl Microsoft-Windows-CAPI2\/Operational \/e:false
To Clear Logs:
wevtutil.exe cl Microsoft-Windows-CAPI2\/Operational
Summary:
Those were the known potential reasons that may have caused scripting delays in the Geo SCADA system. If you believe that none of these are the cause of the script delay, please contact the SE support team for further investigation.
Released for:Schneider Electric UK
This article focuses on troubleshooting delays in script execution when using a reverse proxy over HTTPS. We will explore the potential reasons for these delays and provide possible solutions to resolve the issues.
Understanding the Problem:
In older versions of ClearSCADA and Geo SCADA Expert, a statically compiled implementation of TLS was used for most connections, except for HTTPS connections created by ViewX. These connections, utilized by Mimic scripts running in ViewX to link to the server, relied on Microsoft libraries for HTTP/HTTPS communication. In contrast, newer versions of Geo SCADA Expert now use Microsoft TLS libraries (SCHANNEL) for all underlying connections. This transition enhances security by providing stronger encryption and regular security updates through the operating system. However, it also means that TLS connections must comply with the security configurations dictated by Security Policies, including default, local, and domain-defined settings.
Possible Reasons for Script Delays:
1- Certificate name mismatch.
The certificate name must match the server's name specified in the ViewX client configuration. If not, it will trigger a Windows check for certificate revocation, which is done if the names do not match.
First, launch IIS and verify that the "issued to" field of the self-signed certificate matches the server name:
2- The set-up delay is affecting ViewX client connection to the server.
Verify that in the Configure Connections dialog, the 'Connection Security | Validate Server Certificate' option is disabled. Additionally, ensure that when connecting to Geo SCADA Expert servers, the 'Use Temporary Certificates' option is enabled in the Server Configuration tool under 'System Configuration | Connection Security.'
If the server machines use self-signed certificates, please clear the 'Validate Server Certificate' check box. In this case, the 'Use Temporary Certificates' check box must also be selected in the Connection Security section of the Server Configuration Tool on the server machines.
For more details regarding the connection security settings, please refer to the manual available here.
3- The set-up delay is affecting server-server partner links.
Please verify that the 'Validate Partner Certificate' option is disabled in the Server Configuration under 'System Configuration | Partners | Connection Security' tab. If the other server machines are using self-signed certificates, kindly clear the check box. In this case, ensure that the 'Use Temporary Certificates' check box is selected in the Connection Security section (see previous picture) within the System Configuration branch of the Server Configuration Tool on those other server machines.
Please refer to the manual here for more information regarding the connection security tab.
4- Specific GPO Policies may causing the delays.
Please be aware that altering these policies may affect the connection validation of other applications and may not be suitable for those applications. Additionally, it can impact other client hosts within the domain. Please verify with experts before making any changes to the group policies.
Please note that this issue only occurs on systems that do not have internet access. On systems with internet access, the CRL lookups will succeed and be much faster than 15 seconds (default setting).
The CRL (Certificate Revocation List) lookups are used to check if a digital certificate has been revoked by the Certificate Authority (CA) before establishing a secure connection. When a client tries to connect to a server using an encrypted protocol (like SSL/TLS), it needs to ensure that the server's certificate is still valid and has not been revoked. when the system has internet access, it can quickly retrieve the CRL from certificate authority server. However, if the system lacks internet access, it cannot perform this lookup.
In the absence of internet access, the client may wait for a timeout period while trying to reach the CRL endpoint, leading to a significant delay before it either fails the connection or moves on to other checks.
The suggested changes that have proven effective in addressing the scripting delay are as follows:
Computer Configuration >> Policies >> Windows Settings >> Security Settings >> Public Key Policies >> Certificate Path Validation Settings >> Network Retrieval
Default URL retrieval timeout in seconds = 1
Default path validation cumulative timeout in seconds = 2
Allow issuer certificate retrieval during path validation = Disabled
Computer Configuration >> Policies >> Administrative Templates >> System >> Internet Communications Management >> Internet Communications Settings
Turn off Automatic Root Certificates Update = Enabled
It is possible to identify this issue from Windows events by enabling CAPI2 logging. You can view the Microsoft SCHANNEL path validation checks in the Event Viewer under 'Application and Services | Microsoft | Windows | CAPI2 | Operational' events.
To Enable Logging:
wevtutil.exe sl Microsoft-Windows-CAPI2\/Operational \/e:true
To Save the Log to a File:
wevtutil.exe epl Microsoft-Windows-CAPI2\/Operational filename.elf
To Disable Logging:
wevtutil.exe sl Microsoft-Windows-CAPI2\/Operational \/e:false
To Clear Logs:
wevtutil.exe cl Microsoft-Windows-CAPI2\/Operational
Summary:
Those were the known potential reasons that may have caused scripting delays in the Geo SCADA system. If you believe that none of these are the cause of the script delay, please contact the SE support team for further investigation.
Released for:Schneider Electric UK
