Go is fun

June 1, 2025

At WattOur, we've been building a lot of data processing infra (focusing on PJM but making it extensible enough to include other ISOs in the near future). Our tech stack was quite fragmented-- we started with a lot of Python apps, a familiar base, and have been slowly migrating more and more to Golang. It's been really great so far--a low bar, given anything not Python would ascend to that category. We've been enjoying the simplicity and utility Go offers. I wanted to write about a few challenges we've faced to (1) document these difficulties, acknowledging that there's probably a more elegant solution out there, (2) set a baseline as we invest more into the Go ecosystem, and (3) perhaps field some advice.

Internally, the current focus has been on a microservice that ingests a bunch of data from external APIs, like PJM's DataMiner, at regular intervals and provides the capability to backfill historical data using these APIs. Of course, we need to then expose notifications and an interface for our models to consume this data.

sqlc

As suggested by the brief spec, this service writes and reads up to hundreds of thousands of rows at a time. We chose to use sqlc (a nice breath of fresh air from ORM complexity) but this is not without apparent downsides. Bulk inserts. It sucks, not sure why. With pgx, Postgres' CopyFrom is exposed but unfortunately self destructs on a PK collision. This isn't ideal since some of these intensive inserts are upserts, updating or enhancing existing rows.

It seems the two options for bulk inserts are to either

  1. combine Copy with a temp table and then upsert, or
  2. parse an array of rows, for instance, into per-column slices and upsert directly
  3. naively loop and upsert

I made a quick benchmark on github just to get a feel for the performance gain from COPY. I imagine database people will shriek in fear upon seeing what workloads I used to bench the different methods--but again, just an indicator. The results were more significant than I expected.

Benchmark comparing different bulk sqlc insert methods
Benchmark comparing different bulk sqlc insert methods

Not that we're inserting 1e6 rows (more on the order of 1e4 for a single transaction), but it seems like we'd be very remiss to not use the native Copy. The only downside, of course, is the implementation. Lots of different sqlc functions, duplicated tables and schemas-- it seems to require a whole another layer ontop of sqlc to bundle this in one transaction. Since we're ingesting so much data into so many different tables, that kinda sucks. I've held off on implementing this for now in hopes that a better solution makes itself apparent to me, but not sure.

microservice composition

Amidst a sea (snotgreen) of microservices, be a monolith. If only I had the ability to listen to advice. Maybe WattOur is ambitious, maybe the devs (primarily myself) are stupid--regardless, we split a lot of our core features into a few core microservices. It seems like everything becomes grey the more I learn, so, naturally, there are benefits and drawbacks to this approach.

Good

  • In theory, these core features neatly fall into different domains. Creating interfaces to specify communication between these domains can be pretty neat is done properly. Then, making changing to a particular feature (if delegated correctly) can be constrained to one domain with minimal impact on downstream consumers.
  • Ability to scale the microservices independently in accordance to their very different specifications

Bad

  • Especially in the early stage, many changes are cross cutting and thus involve changes to multiple repos. All of the Docker instances also fry my Macbook.
  • As such, iteration velocity can suffer (which is probably our greatest asset as a newer venture)
  • We don't need to scale yet

sharing rpc protos/stubs

This is something I admittedly haven't had to do before as I usually vie toward monoliths (surprisingly) and don't frequently use RPCs. We currently share RPCs between different Go services and one Python service and thus need stubs for both languages. I'm pretty sure the answer is some combination of git submodules and external registries (like buf), but will need to test some things out (and maybe update here when done). Unfortunately, it's easier to just copy the files for now..

misc

rate limiter

Integrating with external APIs means not abusing said APIs. PJM, for instance, seems to use a sliding window model to track requests. My understanding of their system can be modeled by a bucket of size $n$ (given $n$ requests/minute). If the client takes a instance of the key from the bucket, it will only be added back after 1 minute elapses. This seems different than a token bucket model, where tokens are added at a fixed rate.

When not a PJM member, you're limited to an abysmal 6 requests/minute which becomes very easy to exceed given the row count constraint of 50,000. Thus, I needed some application pacing to emulate the PJM throttling. I learned about the library time/rate but the token bucket model didn't seem to map onto the PJM pacing. Luckily, the solution was quite nice in Go.

func init() {
 /* snip */
 // set up key channel
 consume = make(chan string, len(apiKeys)*PJMRateLimit)
 
 // add all keys initially
 for _, key := range apiKeys {
  for range PJMRateLimit {
   consume <- key
  }
 }
}

func getApiKey(ctx context.Context) (string, error) {
 select {
 case apiKey := <-consume:
  // schedule return automatically in 1m
  time.AfterFunc(time.Minute, func() {
   consume <- apiKey
  })
  return apiKey, nil
 case <-ctx.Done():
  return "", ctx.Err()
 }
}

func getPJM[T any](ctx context.Context, reqUrl string, params map[string]string) ([]T, error) {
 client := &http.Client{Timeout: 30 * time.Second}
 var results []T
 startRow := 1

 fullUrl := fmt.Sprintf("%s/%s", PJMAPIRoot, reqUrl)

 for {
  /* snip */
  
  apiKey, err := getApiKey(ctx)
  if err != nil {
   return nil, fmt.Errorf("fetching api key: %v", err)
  }

  req, err := http.NewRequest(http.MethodGet, u.String(), nil)
  if err != nil {
   return nil, fmt.Errorf("error creating http request: %w", err)
  }
  req.Header.Set("Ocp-Apim-Subscription-Key", apiKey)

  /* snip */
 }
 return results, nil
}

50/50 I'm using the generics inefficiently, though. Next step will be to spawn a bunch of concurrent requests with the varying offsets, but we already have such a significant speed boost by being able to tolerate bursts sized $k \times n$, as opposed to relying on a very naive pacing heuristic (time.Minute / n).

error handling

I've gleaned some discourse throughout the years about Go and error handling, but honestly I haven't minded it so far. It makes me more intentional about my application flows, but at the same time, I can easily envision a future scenario where I start to get frustrated/bogged down by being so pedantic.

deployments on Railway w/ railpack

The last thing I wanted to mention is how awesome Go services are. Coming from Python frameworks like Django/Flask (and, yes, the specs here are a little different, but the comparison still holds), which consume egregious amounts of memory in idle (the avarice), Go is honestly so nice. We went from a Django prototype consuming 700MB idle on Railway to a Go program consuming upwards of 26MB. And RAM is super expensive for some reason, but that's the price of the cloud, I guess.

As alluded to, we've been working on Railway which also has been nice (and the recent migration to their Metal is cool). They recently introduced Railpacks as well, but the documentation is so sparse and it makes creating a custom config very difficult (at least for me). However, it seems super promising, so I'll keep fiddling around with it.