Search for a command to run...
As a dedicated DevOps Engineer, I've immersed myself in the dynamic world of DevOps, sharing my insights through blogs to support the community. I aim to simplify complex processes, empowering both beginners and experts to navigate DevOps with confidence and ease, fostering collective growth in this ever-evolving field.
No comments yet. Be the first to comment.
The WSO2 Product Basic Troubleshooting Blog Series offers comprehensive guidance on resolving common issues encountered while working with WSO2 products. Covering troubleshooting tips, best practices.
When a WSO2 application experiences issues, it is crucial to promptly identify and resolve these problems. This section covers the troubleshooting process specifically for the WSO2 Micro Gateway (MGW), which is a critical component in the handling of...
A practical guide to identifying, measuring, and paying down technical debt in DevOps pipelines and infrastructure
Understanding why modern teams chose containers over bare-metal deployments
If you're running a home server or any Linux machine that's accessible over the internet, password authentication is a ticking time bomb. Automated bots are constantly scanning for SSH servers and attempting brute-force attacks. The solution? SSH key...
If you've tried changing the SSH port on Ubuntu 24.04 and it didn't work, you're not alone. Ubuntu 24.04 handles SSH differently than older versions, and the usual method of editing the config file won't work by itself. In this guide, I'll show you t...
Introduction If you're running a home server and want to access it remotely, you've probably faced this frustrating problem: your ISP keeps changing your public IP address. Every time your router restarts or the IP lease expires, you lose access to y...
After addressing issues potentially arising from the Identity Server or Micro Gateway, the next critical component in your application architecture to troubleshoot is the Micro Integrator (MI). MI plays a pivotal role in integrating different backend systems and services, making it crucial to resolve its issues promptly.
Typical Application Flow:
IS (Identity Server) → MGW (Micro Gateway) → MI (Micro Integrator) → [Your backends (EX; database)]
Note: This flow is illustrative; your actual application setup may differ.
The MI is designed to handle complex integrations and process high volumes of messages efficiently. However, problems such as network connectivity issues, endpoint suspensions,resource limitations and many more errors can arise. Here’s how to systematically identify and resolve these issues:
Symptoms:
Example Command to Identify Errors:
cat wso2carbon.log | grep "error connecting backend" | more
Example Output:
[2023-10-18 09:44:21,155] ERROR {LOGGER_V1} - {"timestamp":"2023-10-18 09:44:21,152","platform":"WSO2","loggerVersion":"1.0","correlationId":"18b4187320b90abc","channel":"SIMAPP","integrationName":"ESB Proxy API","integrationVersion":"1.0.0","messageType":"ESB_PROXY_API","clientMessageId":"339344a6-7ef1-4528-892d-92a8e370cf44","origin":"172.18.139.7","sequenceName":"proxy-api-v1-fault-sequence","appVersion":"1.0.0","appId":"Simbanking","httpMethod":"GET","metaTransportInUrl":"/esb-proxy/1.0.0","contentType":"application/json","context":"/esb-proxy","statusCode":500,"message":"ESB Error, Error connecting to the back end"}
Why This Happened:
Network Issues: There could be a network configuration error such as incorrect proxy settings, DNS issues, or the backend service might be down.
Firewall/Security Configurations: Security appliances (like firewalls) or network policies may be blocking the connection.
Misconfigurations in Service URL: The endpoint URL might be incorrect or the service might not be responding as expected.
How to Solve:
Verify Backend Services: Ensure that the backend services are up and running.
Check Network Configurations: Review network settings, including firewall rules, DNS configurations, and proxy settings.
Service URL: Double-check the service URL and other configurations in your proxy service definition.
Symptoms:
Example Command to Identify Errors:
cat wso2carbon.log | grep "Suspending endpoint" | more
Example Output:
[2024-04-20 17:59:21,260] WARN {org.apache.synapse.endpoints.EndpointContext} - Suspending endpoint : bulk-sms-sender-api-v1-inforbip-bulk-sms-rest-endpoint with address https://9zm33.api.infobip.com/sms/2/text/advanced - current suspend duration is : 2000ms - Next retry after : Sat Apr 20 17:59:23 EAT 2024
Why This Happened:
Service Unavailability: The endpoint might be temporarily unavailable, which triggers the suspension.
Repeated Failures: Endpoint suspension often follows several consecutive failures in reaching the endpoint, as defined in the Synapse configuration.
How to Solve:
Increase Timeout Values: Consider increasing the timeout values and the retry duration in your endpoint configuration.
Monitor Endpoint: Use administrative tools or scripts to monitor the availability of the endpoint.
Review Endpoint Strategy: Implement a more robust error handling and retry mechanism, possibly with dynamic error thresholds.
Symptoms:
Example Command to Identify Errors:
cat wso2carbon.log | grep "dropping message after ENDPOINT_TIMEOUT" | more
Example Output:
[2023-10-27 12:15:08,086] WARN {org.apache.synapse.core.axis2.TimeoutHandler} - Expiring message ID : urn:uuid:6ac77ba7-7525-4abd-9df6-e7ac3420f449; dropping message after ENDPOINT_TIMEOUT of : 30 seconds for Endpoint [proxy-api-v1-service-endpoint], URI : http://172.32.1.57:9090/savvycore/card/cardrequest/createCardRequest, Received through API : proxy-api-v1-api:v1.0.0
Why This Happened:
Resolution Steps:
Adjust Timeout Settings: Increase the endpoint timeout setting to allow more time for slower backend services to respond.
Optimize Backend Performance: Ensure that the backend services are optimized to handle requests within the timeout period.
Error Handling Strategies: Implement comprehensive error handling strategies in the integration logic to manage timeouts gracefully.
Symptoms:
Example Command to Identify Errors:
cat wso2carbon.log | grep "exhaust" | more
Example Output:
Could not get a PassThroughMessageProcessor thread to process the request message. The primary worker pool is exhausted
Why This Happened:
High Concurrency: Too many concurrent requests are being processed, exceeding the capacity of the available worker threads.
Resource Limits: The server's resource limits (CPU, memory, threads) are reached, causing it to deny new requests.
How to Solve:
Increase Thread Pool Size: Increase the size of the worker pool in the MI’s server configurations (axis2.xml or passthru-http.properties).
Optimize Performance: Review the performance of your integrations to ensure efficient use of resources.
Load Balancing: Implement load balancing to distribute requests more evenly across available resources.
Monitoring and Logging: Enhance monitoring and logging practices to get real-time insights and react proactively to issues.
Regular Testing: Regularly test endpoints and backend services to ensure they meet operational standards and performance benchmarks.
Configuration Review: Periodically review and optimize configurations based on operational experience and evolving requirements.
Effectively troubleshooting WSO2 Micro Integrator (MI) is essential for maintaining a robust and reliable application architecture. By systematically identifying and resolving common issues such as backend connectivity errors, endpoint suspensions, message drops, and thread pool exhaustion, you can ensure smooth integrations between services. Key strategies include monitoring logs, optimizing configurations, and enhancing backend performance. Proactive measures, such as increasing timeout values, adjusting thread pool sizes, and implementing load balancing, help mitigate potential issues before they affect application performance. Regular monitoring, logging, and configuration reviews are vital in ensuring that the MI continues to handle complex integrations efficiently in a dynamic and demanding production environment.