I’m hitting some memory limits on this service
The app goes close to the limit of allowed memory (510mb) and then stalls. The container is never killed. The REST API endpoints just timeout after that of the heavy processing requests.
The remedy is to issue a hard reboot of the service.
I don’t know if this is the python gc, or what. But it seems to go to 510 MB before stalling.
What’s a work around? Should I put a watchdog timer that kills the program at 490MB? The CPU seems to continue spinning so there might be enough CPU action to run a sig kill.