Improving the colStats Tool to Process Files Concurrently

Powerful Command-Line Applications in Go — by Ricardo Gerardi (55 / 127)

👈 Tracing Your Tool | TOC | Reduce Scheduling Contention 👉

As you noticed from the tracer output, the tool is processing files sequentially. This isn’t efficient since several files have to be processed. By changing the program to process files concurrently, you can benefit from multiprocessor machines and use more CPUs, generally making the program run faster. The program will spend less time waiting for resources and more time processing files.

One of the main benefits of Go is its concurrency model. Go includes concurrency primitives that allow you to add concurrency to your programs in a more intuitive way. By using goroutines and channels. you can modify the current colStats tool to process several files concurrently by making changes to the run function only. The other functions remain unchanged.

First, add the sync package to the imports section, which provides synchronization types such as the WaitGroup:

You’ll update the run function to process files concurrently by creating a new goroutine for each file you need to process. But first you’ll need to create some channels to communicate between the goroutines. You’ll use three channels: resCh of type chan []float64 to communicate results of processing each file, errCh of type chan error to communicate potential errors, and doneCh of type chan struct{} to communicate when all files have been processed. Define these channels after defining the consolidate variable:

Notice that you’re using an empty struct as the type for the doneCh channel. This is a common pattern since this channel doesn’t need to send any data. It only sends a signal indicating the processing is done. By using the empty struct, the program doesn’t allocate any memory for this channel.

Next, define a variable wg of type sync.WaitGroup. The WaitGroup provides a mechanism to coordinate the…

Related posts:

What if today was your last day? What if there was no more? What if it all ended here? What would be left? What would be the conclusion of your life? What if everything you hold dear, vanished? What…

As I sat there in my wheelchair looking at Trinity I noticed something, a pack of cards on her bed table. “Huh. Odd.” I said to myself. I wheeled over beside her and I positioned myself so I could…

Im trying to trade in my 2008 chevy hhr for a 2009 Honda Ridgeline which is 33,000 and I am trying to get a auto loan too. I was trying to get a 15,000 to 20,000 auto loan. Do i have to pay both the…