Memory Usage Tracking In Golang

Go tool pprof

Go’s profiling tool is useful for identifying and correcting potential bottlenecks. The tool can be used in a variety of modes, including

• Heap — memory allocations of live objects
• Threadcreate — stack traces that led to the creation of new OS threads
• Goroutine — stack traces of all current goroutines
• Allocs — a sampling of memory allocations of live objects
• Block — stack traces that led to blocking on synchronization primitives
• Mutex — stack traces of holders of contended mutexes

The options are mutually exclusive, so you can only profile one option at any given time.

The PPROF server

Profiling can be enabled over HTTP which gives access to web based endpoints.

To start the pprof server,

Memory profiling

Investigating the memory errors for our service described above, we would consistently see a rise in memory which is never released.

Writing the profile to a file

Using the heap endpoint after enabling the pprof server for memory profiling, we are able to hit the heap profile to get a snapshot of the current memory. Writing this information automatically to a file gives the ability to deploy the service and have it collect samples on its own. If we were running the service locally, we could use the curl command then dump the results to a file as well.

curl http://localhost:8080/debug/pprof/heap > heap.0.pprof
sleep 30
curl http://localhost:8080/debug/pprof/heap > heap.1.pprof
sleep 30
curl http://localhost:8080/debug/pprof/heap > heap.2.pprof
sleep 30
curl http://localhost:8080/debug/pprof/heap > heap.3.pprof

The advantage of deploying the service with the profiler enabled is that we can get real samples without having to reproduce the conditions locally. In fact, I ran the service over the weekend with the profiler enabled and the CPU and memory footprints are about the same as with the profiler disabled.

Analyzing with pprof

To start the pprof tool for analyzing a file,

go tool pprof heap.out

Where “heap.out” is the name of the file you want to analyze. It is also possible to diff two files

go tool pprof --base heap.1592551172883.pprof heap.1592835572964.pprof

Here I’m doing a diff on an early file, and the last file which was generated.

In Short, if you want to check out the memory usage in golang after and before some functions or some go routines you can easily do that by doing so…

func (w *Queue) PrintMemUsage(queue string) {
var m runtime.MemStats
runtime.ReadMemStats(&m)
// For info on each, see: https://golang.org/pkg/runtime/#MemStats
w.logger.Debug(fmt.Sprintf("Alloc = %v MiB , TotalAlloc = %v MiB , Sys = %v MiB , NumGC = %v\n", bToMb(m.Alloc), bToMb(m.TotalAlloc), bToMb(m.Sys), m.NumGC), zap.Any("queue", queue))
}
func bToMb(b uint64) uint64 {
return b / 1024 / 1024
}

If you found this helpful, make sure to show your support with a clap, and if you want to help others in their projects, a share would be greatly appreciated! Let me know what you think about this! Happy Learning!