Bug 65213 - Endurance Testing using jmeter stopped abruptly in distributed environment
Summary: Endurance Testing using jmeter stopped abruptly in distributed environment
Alias: None
Product: JMeter
Classification: Unclassified
Component: Main (show other bugs)
Version: 5.2.1
Hardware: Other Linux
: P2 critical with 1 vote (vote)
Target Milestone: JMETER_5.2.1
Assignee: JMeter issues mailing list
Depends on:
Reported: 2021-03-31 13:12 UTC by Amit Agrawal
Modified: 2021-06-06 13:54 UTC (History)
1 user (show)

Test Plan (203.58 KB, image/png)
2021-03-31 13:17 UTC, Amit Agrawal
jmeter-server.log file (51.24 KB, text/plain)
2021-03-31 13:20 UTC, Amit Agrawal

Note You need to log in before you can comment on or make changes to this bug.
Description Amit Agrawal 2021-03-31 13:12:30 UTC
I am using Azure Kubernetes Service(AKS) to deploy JMeter infrastructure. Using JMeter - 5.2.1 version and Kubernetes version - 1.19.6.

I am running 24 hrs endurance test using JMeter with ThroughputShapingTimer. My test stopped abruptly some time after 6 hrs and some time 8 hrs.

I checked jmeter-server.log file and getting warning - WARN k.a.j.t.VariableThroughputTimer: No free threads left in worker pool.
No error observed in jmeter.log from jmeter slave.

I used 400 threads and then 800 threads and then tried 1100 threads still getting  - No free threads left in worker pool.

I increased jmeter slave and master memory from 2GB to 6GB and 4 cores of CPU still issue persists.

I checked kubernetes PODS, they were not restarted so assuming there is Out of Memory concern. 
I ran with 2 slaves and 1 master also.

Please advise how to overcome this issue.

Following attachments are provided.
1. Test Plan
2. Jmeter-server.log file.
Comment 1 Amit Agrawal 2021-03-31 13:17:24 UTC
Created attachment 37793 [details]
Test Plan
Comment 2 Amit Agrawal 2021-03-31 13:20:23 UTC
Created attachment 37794 [details]
jmeter-server.log file
Comment 3 Felix Schumacher 2021-06-06 13:54:27 UTC
The VariableThroughputTimer is a third party plugin. Can you reproduce the problem without it?

I am guessing that the message comes from an overloaded system that you are testing and all threads are busy with doing a sample already.

I would do a thread dump, when JMeter "works" and some, when you think it stopped working.