Troubleshooting the E3 Bridge
The E3 Bridge is a WPF application similar to TimeLine E3 but hosted in a PowerBuilder ActiveX control inside TimeLine Enterprise E2.
Several types of issues can occur at runtime, sometimes the Bridge crashes itself and sometimes there are other problems due to external stuff involved in the execution of the application, like 3rd party controls and components, application execution trust and permissions to access resources, etc.
Debugging a production environment is not possible mainly because of two reasons: lack of a debugger installed on the production machine and missing debug libraries. The alternatives to debug are:
- Using the available configuration keys to work around the issues
- Reproducing the issue on development machines where there's a debugger and the source code
- Use log files to trace down the error
- Collect memory dumps to be analyzed offline
Locating the cause of the error
Locating the cause of the issue depends very much on intuition depending on the behavior of the application when it produces the issue. Errors and crashes can vary quite much depending of many factors, from system harware and software configuration, updates of all tiers from operating system to TimeLine applications, database version and patch level, etc.
To categorize the errors that can occur, the main ones are:
- hangs: application UI is unresponsive or the app process is not responding
- crashes: the application window completely disappears as well as its process
- handled errors that don’t make the application crash: they should not produce application instability
- warnings: not considered errors although users often tend to call them crashes
Application hang
When an application hang occurs, there are a couple of things to do. First, check Task Manager. If the process is unresponsive but is not listed as "not responding" in Task Manager, then the application may be functioning normally but is waiting for data - possibly from a network resource. If you are experiencing an issue as a result of a network bottleneck or network resource failure, the symptom will most likely be evident on multiple client systems - where all their applications waiting on that resource, or set of resources become unresponsive at the same time. In some instances, the application may be functioning quite normally, however it is actually waiting for user input, but the user input box has somehow been hidden behind other open windows. If however, the application is listed as "Not Responding" in Task Manager, then it may truly be hung, but keep in mind are some scenarios where applications that are waiting for network resources may also be listed as "not responding".
In the event of a hang, we may need to capture a memory dump (or a number of dumps) of the application while it is in its unresponsive state. You can do this by creating a dump of the process from within Task Manager. To do this, highlight the process, right-click and select the "Create Dump File" option.
Application crash
An application crash is defined as a complete vanish of the application process and its main window. In this case, the only solution to profile it is to read the log files or to inspect memory dump files if available.
Check the Windows event log (in Event Viewer) for application crashes, you will eventually find automatically generated memory dump files. If the error cannot be determined from the logs, then collecting crash dumps manually might help.
All these types of issues will be mitigated using different approaches. For instance, when the application crashes, its process dissapperas from the memory most likely together with its main window so that the user has no possibility to further investigate it. If there is an UI app hang, then the user can at least collect a process memory dump for offline analysis. However, checking the error logs at different tiers remain the last resort in many cases.
Configuring Bridge features to work around problems
Sometimes problems in the E3 Bridge occur becasue of 3rd party components used by TimeLine internally such as UI controls from DevExpress, TextControl or others. While other times they were caused by the hardware like display adapter being used.
Below is a list of TimeLine components that can be configured to make the application more compatible with the runtime system or to profile the defect si that it can be solved in a later patch.
Dms preview control
Preview of dms documents is done using several controls depending on the type of the document. For instance, a browser control is used to preview text files, html, emails and text-based formats in general. Preview of Microsoft Office documents is delegated to the operating system via a special Windows interface that Office also implements.
For pdf preview, we can configure much more. Visualizing content of pdf documents is normally done using a control from DevExpress but it can be configured in tl.ini as of version 14 like this:
Edit tl.ini and add the key PdfPreviewer to [TLSYS] section, having the following possible values:
- DevExpress: use a pdf previewer control from DevExpress that is embedded in the E3 bridge. This is the default option if the ini key is missing. But the problem is that often, on some systems it crashes the whole process due to internal DevExpress errors (eg. out of memory erorr).
- System: use the default pdf application that is registered in Windows. The application must support preview feature to work because the application will not be opened to perform preview but it will be used by an Windows interface in the background via COM Server to deliver the preview content.
- None: this option completely disables the pdf preview in the Bridge
Configure the WPF render mode
The internal architecture of WPF has two rendering pipelines, hardware and software. To troubleshoot rendering issues, it is worth to switch the Bridge to use software rendering, this to avoid potential issues coming from the 3rd party hardware display drivers.
This can be done in TL.ini as of version 14 using the key [TLSYS].WpfRenderMode. Possible values are software and default.
Using value software determines WPF to use the software rendering pipeline which is entirely CPU bound. WPF takes advantage of the SSE and SSE2 instruction sets in the CPU to implement an optimized, fully-featured software rasterizer. Fallback to software is seamless any time application functionality cannot be rendered using the hardware rendering pipeline.
Option default means WPF decides what rendering pipeline to use based on system capabilities.
Bound the E2 executable to specific GPU
In practice there were cases when the PowerBuilder virtual machine runnig E2 crashed because of some out-of-memory errors coming from the UI render side when using NVidia GPUs. If your system uses such a GPU, the recommendation is to check if its driver is up to date. If it is but the issue still occurs, please try running your app on the graphical adapter that is integrated in the CPU instead of the Nvidia chip. You can configure this in the Windows display settings. Open SystemSettings -> System -> Display -> GraphicsSettings, browse to your TL150.exe/PBxx.exe and then configure it to use the PowerSaving GPU:
Disable hardware acceleration for WPF or force WPF into a simulated hardware rendering mode
It is possible to completely disable the hardware acceleration for all WPF applications, this for troubleshooting, debugging, and product support purposes. When you see rendering artifacts or strange UI issues in a WPF application, try turning off hardware acceleration. If the issue disappears, the problem might be with your video driver.
To disable hardware acceleration for WPF, simply set the registry key
HKEY_CURRENT_USER\SOFTWARE\Microsoft\Avalon.Graphics\DisableHWAcceleration
to DWORD value 1.
In a RDP/TerminalServer arhitecture, it might be useful to force WPF into a simulated hardware rendering mode for debugging: WPF goes into hardware mode, but uses the Microsoft Direct3D reference software rasterizer, d3dref9.dll, instead of an actual hardware device.
Clear the WFP font cache or completely disable the service
The WPF font cache optimizes performance of Windows WPF applications by caching commonly used font data. WPF applications will start this service if it is not already running. It can be disabled, though doing so will degrade the performance of WPF applications.
Procedure to fix it:
- stop all wpf applications
- stop the windows service Windows Presentation Foundation Font Cache 3.0.0.0
- Delete the old Fontcache3.0.0.0.dat file located at %windir%\ServiceProfiles\LocalService\AppData\Local
- start the service and the application back
Note: Disabling the WPF font cache service completely might make sense if it makes troubles. This will not affect the application functionality but it will reduce its performance. This solution can be taken into consideration at least to profile if the WFP font cache is the source of the cpu load issue.
Temporarily disabling the antivirus
Sometimes the AV interferes so severely that it freezes the application and makes it unresponsive.
Reading log files
Windows logging
Windows logs can be read in the EventViewer tool. This is most likely the place where to find information about application unhandled exceptions or application crashes. It is possible to also find memory dumps if the Windows Error Reporting Windows feature is enabled on the system.
TimeLine logging
TimeLine provides logging support at multiple tiers: client side, client framework, server and database. All these logs might help to narrow down the issue.
The E3 bridge produces 2 log files on the client:
- tlframework.log file saved in the same foder where tl.ini is. Logging needs to be activated in TL.ini, section [TLSYS], key LoggingActive = True
- tl.log file saved in the E3 install folder on the client machine. This is a debug log of the E3 Framework. It is always active and collects some Framework specific messages.
The TimeLine Server can produce its own log files that sometimes are useful. Logging can be temporarily activated in the server console:
The logs are saved in the file tls.log at the server install path.
On server there is also a dataset error log to be found at path \Logs\Sql\etc.
Collecting and analyzing memory dump files
Collecting memory dumps
This can be doen using several tools and procedures at your choice, below a few options:
Sysinternals Process Explorer tool
Find it at https://docs.microsoft.com/en-us/sysinternals/
Sysinternals ProcDump tool
ProcDump is part of the SysInternals toolkit, which is a set of extremely useful tools for Windows development. ProcDump itself is a command line tool for creating dumps. It can generate dumps on demand or by monitoring a program and create a dump on a crash or a hang.
A simple use case to immediately create a dump with full memory is:
procdump -ma [process ID]
procdump -ma [name of executable, for example "notepad"]
This creates a dump file in your working directory named: [process name]_[date]_[time].dmp.
You can monitor for things like memory usage, hung programs, CPU usage, first-chance exceptions and create a dump on demand. For example, this captures a dump when myprogram.exe is hung to a file named dumpfile.dmp.
procdump -h myprogram.exe dumpfile.dmp
Details can be found here: https://docs.microsoft.com/en-us/sysinternals/downloads/procdump
DebugDiag Collection tool
The tool is able to capture application crashes, call stacks etc to be used for detecting performance problems, deadlocks, .net memory allocation issues, as well as native memory and handle leaks.
Windows Error Reporting tool from Windows (WER)
Windows Error Reporting (WER) is a flexible event-based feedback infrastructure designed to gather information about the hardware and software problems that Windows can detect, report the information to Microsoft, and provide users with any available solutions. Windows provides crash, no response, and kernel fault error reporting by default without requiring changes to your application. Applications instead use the WER API to generate error reports for application-specific issues that are not related to crashes, non-responses, or kernel faults.
Please find more details at https://docs.microsoft.com/en-us/windows/win32/wer/about-wer.
You can have Windows automatically generate dump files on program crash. This is done with Windows Error Reporting.
Configuring WER
- Make sure the WER windows service is up and running
- Run Registry Editor (regedit.exe)
- Open or create the key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps
- Add a key with the same name as your program executable
- Add the following keys:
DumpFolder (REG_EXPAND_SZ) – folder where dump should be created
Optionally DumpCount (REG_DWORD) – maximum dumps created before old dumps will be overwritten. The default is 10.
DumpType (REG_DWORD) – minidump (1) or mull dump (2). Use Full dump for .NET.
The registry should look like this:
After WER is configured, reproduce the crash so that the dump file will be saved at the configured location.
Analyzing memory dumps
Creating Dumps is the easy part. The real challenge is debugging them and understanding the problem. For this, please send the dump files to the TimeLine development team for analysis.
Last updated
