GigE Cameras experience Heartbeat timeout after periods of inactivity #863
Description
Describe the issue:
Four cameras running off of software triggers and GrabStrategy_LatestImageOnly are locking out after periods of inactivity. The exact amount of time doesn't seem to be consistent, but maybe after an hour? If I send a software trigger after this point, it throws an exception.
The cameras are a2A4504-5gmBAS. They are plugged into a PoE switch, and running off an HP Elite Mini running Windows 11. I recently migrated off of a laptop running windows 10 where they worked fine for two years.
For whatever reason, Pylon seems to have waited too long to poll the cameras for heartbeat- about 3.6 seconds after an hour of regular polling intervals of less than a second. I will increase the polling interval from 3 seconds to see if that helps, but it does not address a root cause.
Here are the Wireshark packets from around the time when it occurs:
No. Time Source Destination Protocol Length Info
320657 16:06:24.956000 192.168.1.3 192.168.1.2 GVCP 60 < WRITEREG_ACK (Success)
320658 16:06:24.956000 192.168.1.6 192.168.1.2 GVCP 60 < WRITEREG_ACK (Success)
320671 16:06:25.242776 192.168.1.2 192.168.1.4 GVCP 58 > WRITEREG_CMD [Heartbeat timeout] Value=0x00000BB8
320672 16:06:25.243781 192.168.1.4 192.168.1.2 GVCP 60 < WRITEREG_ACK (Success)
320676 16:06:25.319977 192.168.1.2 192.168.1.5 GVCP 58 > WRITEREG_CMD [Heartbeat timeout] Value=0x00000BB8
320678 16:06:25.321141 192.168.1.5 192.168.1.2 GVCP 60 < WRITEREG_ACK (Success)
320696 16:06:25.707195 192.168.1.2 192.168.1.3 GVCP 58 > WRITEREG_CMD [Heartbeat timeout] Value=0x00000BB8
320697 16:06:25.707195 192.168.1.2 192.168.1.6 GVCP 58 > WRITEREG_CMD [Heartbeat timeout] Value=0x00000BB8
320698 16:06:25.708241 192.168.1.3 192.168.1.2 GVCP 60 < WRITEREG_ACK (Success)
320699 16:06:25.708347 192.168.1.6 192.168.1.2 GVCP 60 < WRITEREG_ACK (Success)
320717 16:06:25.994340 192.168.1.2 192.168.1.4 GVCP 58 > WRITEREG_CMD [Heartbeat timeout] Value=0x00000BB8
320718 16:06:25.995323 192.168.1.4 192.168.1.2 GVCP 60 < WRITEREG_ACK (Success)
320720 16:06:26.072403 192.168.1.2 192.168.1.5 GVCP 58 > WRITEREG_CMD [Heartbeat timeout] Value=0x00000BB8
320721 16:06:26.073469 192.168.1.5 192.168.1.2 GVCP 60 < WRITEREG_ACK (Success)
320731 16:06:29.319731 192.168.1.2 192.168.1.3 GVCP 58 > WRITEREG_CMD [Heartbeat timeout] Value=0x00000BB8
320730 16:06:29.319795 192.168.1.2 192.168.1.6 GVCP 58 > WRITEREG_CMD [Heartbeat timeout] Value=0x00000BB8
320732 16:06:29.319837 192.168.1.2 192.168.1.4 GVCP 58 > WRITEREG_CMD [Heartbeat timeout] Value=0x00000BB8
320733 16:06:29.319865 192.168.1.2 192.168.1.5 GVCP 58 > WRITEREG_CMD [Heartbeat timeout] Value=0x00000BB8
320745 16:06:29.320424 192.168.1.6 192.168.1.2 GVCP 60 < WRITEREG_ACK (Access Denied) (Failed)
320746 16:06:29.320424 192.168.1.4 192.168.1.2 GVCP 60 < WRITEREG_ACK (Access Denied) (Failed)
320747 16:06:29.320506 192.168.1.5 192.168.1.2 GVCP 60 < WRITEREG_ACK (Access Denied) (Failed)
320748 16:06:29.320506 192.168.1.3 192.168.1.2 GVCP 60 < WRITEREG_ACK (Access Denied) (Failed)
Reproduce the code example:
# Camera setup:-----------------------------------------------------
tlf = pylon.TlFactory.GetInstance()
devices = tlf.EnumerateDevices()
for d in devices: # List the devices by Model name and serial number
print(d.GetModelName(), d.GetSerialNumber())
print(d.GetFriendlyName())
cam1 = pylon.InstantCamera(tlf.CreateDevice(devices[0]))
cam2 = pylon.InstantCamera(tlf.CreateDevice(devices[2]))
cam3 = pylon.InstantCamera(tlf.CreateDevice(devices[3]))
cam4 = pylon.InstantCamera(tlf.CreateDevice(devices[1]))
cam1.RegisterConfiguration(pylon.SoftwareTriggerConfiguration(), pylon.RegistrationMode_ReplaceAll, pylon.Cleanup_Delete )
cam2.RegisterConfiguration(pylon.SoftwareTriggerConfiguration(), pylon.RegistrationMode_ReplaceAll, pylon.Cleanup_Delete )
cam3.RegisterConfiguration(pylon.SoftwareTriggerConfiguration(), pylon.RegistrationMode_ReplaceAll, pylon.Cleanup_Delete )
cam4.RegisterConfiguration(pylon.SoftwareTriggerConfiguration(), pylon.RegistrationMode_ReplaceAll, pylon.Cleanup_Delete )
cam1.Open()
cam2.Open()
cam3.Open()
cam4.Open()
print("cams Opened")
# set exposure times for cameras
camExposure = 20000.0
ce = str(camExposure)
# camera.Gain.SetValue(6);
# camera.Gamma.SetValue(1.2);
cam1.ExposureTime.SetValue(camExposure)
cam2.ExposureTime.SetValue(camExposure)
cam3.ExposureTime.SetValue(camExposure)
cam4.ExposureTime.SetValue(camExposure)
cam1.Gamma.SetValue(0.8)
cam2.Gamma.SetValue(0.8)
cam3.Gamma.SetValue(0.8)
cam4.Gamma.SetValue(0.75)
# Start grabbing from all cameras
cam1.StartGrabbing(pylon.GrabStrategy_LatestImageOnly)
cam2.StartGrabbing(pylon.GrabStrategy_LatestImageOnly)
cam3.StartGrabbing(pylon.GrabStrategy_LatestImageOnly)
cam4.StartGrabbing(pylon.GrabStrategy_LatestImageOnly)
#Image acquire-----------------------------------------------
t1 = time.perf_counter()
# Retrieve the results one at a time
if cam1.WaitForFrameTriggerReady(500, pylon.TimeoutHandling_ThrowException):
cam1.ExecuteSoftwareTrigger()
t2 = time.perf_counter()
time.sleep(0.5) #I hate them, but the sleep commands ensure the trailing lb's are in the sweet spots
if cam2.WaitForFrameTriggerReady(500, pylon.TimeoutHandling_ThrowException):
cam2.ExecuteSoftwareTrigger()
t3 = time.perf_counter()
time.sleep(0.5)
if cam3.WaitForFrameTriggerReady(500, pylon.TimeoutHandling_ThrowException):
cam3.ExecuteSoftwareTrigger()
t4 = time.perf_counter()
time.sleep(0.5)
if cam4.WaitForFrameTriggerReady(500, pylon.TimeoutHandling_ThrowException):
cam4.ExecuteSoftwareTrigger()
t5 = time.perf_counter()
print("trig1 = " + str(t2 - t1))
print("trig2 = " + str(t3 - t2 - 0.5))
print("trig3 = " + str(t4 - t3 - 0.5))
print("trig4 = " + str(t5 - t4 - 0.5))
with cam1.RetrieveResult(2000) as result:
img1 = result.GetArray()
#img1 = cv2.resize(img1, (1000, 1000))
print("cam1 got the image")
photos=[undistort(img1)]
if carLen > 1:
with cam2.RetrieveResult(2000) as result:
img2 = result.GetArray()
#fix = undistort(undistort(img2))
photos.append(undistort(img2))
if carLen > 2:
with cam3.RetrieveResult(2000) as result:
img3 = result.GetArray()
#img3 = cv2.resize(img3, (1000, 1000))
photos.append(undistort(img3))
if carLen > 3:
with cam4.RetrieveResult(2000) as result:
img4 = result.GetArray()
#img4 = cv2.resize(img4, (1000, 1000))
photos.append(undistort(img4))Error message:
Is your camera operational in Basler pylon viewer on your platform
Yes
Hardware setup & camera model(s) used
4 cameras:
a2A4504-5gmBAS
64-bit operating system, x64-based processor
PC:
HP Elite Mini
12th Gen Intel(R) Core(TM) i5-12500T (2.00 GHz)
Windows 11, Version 24H2 (OS Build 26100.7462)
Runtime information:
python: 3.10.4 (tags/v3.10.4:9d38120, Mar 23 2022, 23:13:41) [MSC v.1929 64 bit (AMD64)]
platform: win32/AMD64/10
pypylon: 4.2.0 / 10.2.1.471