Since November 24 we’ve started getting very frequent Server Unhealthy messages with the errors Exited with status 1 or Out of memory despite inspecting the latest commit since this started happening and seeing that it doesn’t have any memory leaks making the service inaccessible, after checking some similar topics here with a similar issue, we’ve upgraded the plan from Standard Plus (3GB RAM) to Pro (4GB RAM) and added --max-old-space-size flag, and that solved the issue temporarily on the weekend but it started happening again on Monday and I’ve reached to the chat support on Thursday and they asked for the service ID which I provided on Friday morning but haven’t heard back from them since then.
Besides upgrading the plan we did some memory profiling for the app locally and didn’t show any usage near the 4GB we have in our plan so we are not sure if this is related to the server or app and would like some pointers on what to do next.
This is the service ID in question srv-c042e5qli6carbd3tev0 and below is the log we got before we’ve upgraded to the Pro plan and added the --max-old-space-size flag.
Nov 24 05:41:25 PM <--- Last few GCs --->
Nov 24 05:41:25 PM
Nov 24 05:41:25 PM [66:0x4d85560] 298840 ms: Scavenge 762.7 (1291.6) -> 757.0 (1301.3) MB, 8.9 / 0.0 ms (average mu = 0.989, current mu = 0.973) allocation failure
Nov 24 05:41:25 PM [66:0x4d85560] 298855 ms: Scavenge 764.9 (1301.3) -> 762.2 (1306.1) MB, 5.2 / 0.0 ms (average mu = 0.989, current mu = 0.973) task
Nov 24 05:41:25 PM [66:0x4d85560] 298874 ms: Scavenge 770.7 (1306.1) -> 766.7 (1313.3) MB, 5.8 / 0.0 ms (average mu = 0.989, current mu = 0.973) allocation failure
Nov 24 05:41:25 PM
Nov 24 05:41:25 PM
Nov 24 05:41:25 PM <--- JS stacktrace --->
Nov 24 05:41:25 PM
Nov 24 05:41:25 PM FATAL ERROR: MarkCompactCollector: young object promotion failed Allocation failed - JavaScript heap out of memory
Nov 24 05:41:25 PM 1: 0xa222f0 node::Abort() [node]
Nov 24 05:41:25 PM 2: 0x96411f node::FatalError(char const*, char const*) [node]
Nov 24 05:41:25 PM 3: 0xb97f1e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
Nov 24 05:41:25 PM 4: 0xb98297 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
Nov 24 05:41:25 PM 5: 0xd52fd5 [node]
Nov 24 05:41:25 PM 6: 0xd83bbe v8::internal::EvacuateNewSpaceVisitor::Visit(v8::internal::HeapObject, int) [node]
Nov 24 05:41:25 PM 7: 0xd8fbf6 v8::internal::FullEvacuator::RawEvacuatePage(v8::internal::MemoryChunk*, long*) [node]
Nov 24 05:41:25 PM 8: 0xd7bd8f v8::internal::Evacuator::EvacuatePage(v8::internal::MemoryChunk*) [node]
Nov 24 05:41:25 PM 9: 0xd7c008 v8::internal::PageEvacuationTask::RunInParallel(v8::internal::ItemParallelJob::Task::Runner) [node]
Nov 24 05:41:25 PM 10: 0xd6e8e9 v8::internal::ItemParallelJob::Run() [node]
Nov 24 05:41:25 PM 11: 0xd91b50 void v8::internal::MarkCompactCollectorBase::CreateAndExecuteEvacuationTasks<v8::internal::FullEvacuator, v8::internal::MarkCompactCollector>(v8::internal::MarkCompactCollector*, v8::internal::ItemParallelJob*, v8::internal::MigrationObserver*, long) [node]
Nov 24 05:41:25 PM 12: 0xd923ec v8::internal::MarkCompactCollector::EvacuatePagesInParallel() [node]
Nov 24 05:41:25 PM 13: 0xd925b5 v8::internal::MarkCompactCollector::Evacuate() [node]
Nov 24 05:41:25 PM 14: 0xda45b1 v8::internal::MarkCompactCollector::CollectGarbage() [node]
Nov 24 05:41:25 PM 15: 0xd60878 v8::internal::Heap::MarkCompact() [node]
Nov 24 05:41:25 PM 16: 0xd62368 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
Nov 24 05:41:25 PM 17: 0xd657ac v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
Nov 24 05:41:25 PM 18: 0xd2af4d v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [node]
Nov 24 05:41:25 PM 19: 0xd24dd4 v8::internal::FactoryBase<v8::internal::Factory>::AllocateRawWithImmortalMap(int, v8::internal::AllocationType, v8::internal::Map, v8::internal::AllocationAlignment) [node]
Nov 24 05:41:25 PM 20: 0xd27121 v8::internal::FactoryBase<v8::internal::Factory>::NewRawTwoByteString(int, v8::internal::AllocationType) [node]
Nov 24 05:41:25 PM 21: 0xe4afb2 v8::internal::JsonParser<unsigned short>::MakeString(v8::internal::JsonString const&, v8::internal::Handle<v8::internal::String>) [node]
Nov 24 05:41:25 PM 22: 0xe4d20d v8::internal::JsonParser<unsigned short>::ParseJsonValue() [node]
Nov 24 05:41:25 PM 23: 0xe4db8f v8::internal::JsonParser<unsigned short>::ParseJson() [node]
Nov 24 05:41:25 PM 24: 0xc42f55 [node]
Nov 24 05:41:25 PM 25: 0xc43656 v8::internal::Builtin_JsonParse(int, unsigned long*, v8::internal::Isolate*) [node]
Nov 24 05:41:25 PM 26: 0x1423359 [node]