神刀安全网

Go best practices, six years in

( This article was originally a talk at QCon London 2016. Slides and video coming soon. )

In 2014, I gave a talk at the inaugural GopherCon titled Best Practices in Production Environments . We were early adopters at SoundCloud , and by that point had been writing, running, and maintaining Go in production in one form or another for nearly 2 years. We had learned a few things, and I tried to distill and convey some of those lessons.

Since then, I’ve continued working in Go full-time, later on the activities and infrastructure teams at SoundCloud, and now at Weaveworks , on Weave Scope and Weave Mesh . I’ve also been working hard on Go kit , an open-source toolkit for microservices. And all the while, I’ve been active in the Go community, meeting lots of developers at meetups and conferences throughout Europe and the US, and collecting their stories—both successes and failures.

With the 6th anniversary of Go’s release in November of 2015, I thought back to that first talk. Which of those best practices have stood the test of time? Which have become outmoded or counterproductive? Are there any new practices that have emerged? In March, I had the opportunity to give a talk at QCon London where I reviewed the best practices from 2014 and took a look at how Go has evolved in 2016. Here’s the meat of that talk.

I’ve highlighted the key takeaways as linkable Top Tips.

Top Tip— Use Top Tips to level up your Go game.

And a quick table of contents…

  1. Development environment
  2. Logging and instrumentation
  3. Dependency management

Development environment original

Go has development environment conventions centered around the GOPATH. In 2014 I advocated strongly for a single global GOPATH. My positioned has softened a bit. I still think that’s the best idea, all else equal, but depending on your project or team, other things may make sense, too.

If you or your organization produces primarily binaries, you might find some advantages with a per-project GOPATH. There’s a new tool, gb , from Dave Cheney and contributors, which replaces the standard go tooling for this use-case. A lot of people are reporting a lot of success with it.

Some Go developers use a two-entry GOPATH, e.g. $HOME/go/external:$HOME/go/internal . The go tool has always known how to deal with this: go get will fetch into the first path, so it can be useful if you need strict separation of third-party vs. internal code.

One thing I’ve noticed some developers forget to do: put GOPATH/bin into your PATH. This allows you to easily run binaries you get via go get, and makes the (preferred) go install mechanism of building code easier to work with. No reason not to do it.

Top Tip— Put $GOPATH/bin in your $PATH, so installed binaries are easily accessible.

Regarding editors and IDEs, there’s been a lot of steady improvement. If you’re a vim warrior, life has never been better: thanks to the tireless and extremely capable efforts of Fatih Arslan , the vim-go plugin is in an absolutely exceptional state, best-in-class. I’m not as familiar with emacs, but Dominik Honnef’s go-mode.el is still the big kahuna there.

Moving up the stack, lots of folks are still using and having success with Sublime Text + GoSublime . And it’s hard to beat the speed. But more attention seems to be paid lately to the Electron-powered editors. Atom + go-plus has many fans, especially those developers that have to frequently switch languages to JavaScript. The dark horse has been Visual Studio Code + vscode-go , which, while slower than Sublime Text, is noticably faster than Atom, and has excellent default support for important-to-me features, like click-to-definition. I’ve been using it daily for about half a year now, after being introduced to it by Thomas Adam . Lots of fun.

In terms of full IDEs, the purpose-built LiteIDE has been receiving regular updates and certainly has its share of fans. And the IntelliJ Go plugin has been consistently improving as well.

Repository structure original

We’ve had a lot of time for projects to mature, and some clear patterns have emerged. How you structure your repo depends on what your project is . First, if your project is private or internal to your company, go nuts: have it represent its own GOPATH, use a custom build tool, whatever makes you happy and productive.

However, if your project is public (i.e. open-source), the rules get a little stricter. You should play nice with go get, because that’s still how most Go developers will want to consume your work.

Ideal structure depends on the type of your artifacts. If your repo is exclusively a binary or a library, then make sure consumers can go get or import it by its base path. That is, put the package main or primary importable resource in github.com/name/repo, and use subdirectories for helper packages.

If your repo is a combination of binaries and libraries, then you should determine which is the primary artifact, and put that in the root of the repo. For example, if your repo is primarily a binary, but can also be used as a library, then you’ll want to structure it like this:

github.com/peterbourgon/foo/     main.go      // package main     main_test.go // package main     lib/         foo.go      // package foo         foo_test.go // package foo 

One useful thing is to name the package in the lib/ subdirectory for the library rather than the directory, in this case making it package foo rather than package lib. This is an exception to an otherwise pretty strict Go idiom, but in practice, it’s very friendly for consumers. One great repo that’s laid out this way is tsenart/vegeta , an HTTP load testing tool.

Top Tip— If your repo foo is primarily a binary, put your library code in a lib/ subdir, and call it package foo.

If your repo is primarily a library, but it also includes one or more binaries, then you’ll want to structure it like this:

github.com/peterbourgon/foo     foo.go      // package foo     foo_test.go // package foo     cmd/         foo/             main.go      // package main             main_test.go // package main 

You’ll invert the structure, putting the library code at the root, and making a cmd/foo/ subdirectory to hold the binary code. The cmd/ intermediary is nice for two reasons: the Go tooling automatically names binaries after the directory of their package main, so it allows us to give them the best possible names without potentially conflicting with other packages in the repo; and it makes it unambiguous to your consumers what they’re getting when they go get a path with /cmd/ in the middle. The build tool gb is laid out this way.

Top Tip— If your repo is primarily a library, put your binaries in separate subdirectories under cmd/.

The idea here is to optimize for consumers: make it easy to consume the most common form of your project. This abstract idea, a focus on consumers, is I think part of the Go ethos.

Formatting and style original

Things have stayed largely the same here. This is one area that Go has gotten quite right, and I really appreciate the consensus in the community and stability in the language.

The Code Review Comments are great, and should be the minimum set of critera you enforce during code review. And when there are disputes or inconsistencies in names, Andrew Gerrand’s idiomatic naming conventions are a great set of guidelines.

Top Tip— Defer to Andrew Gerrand’s naming conventions .

And in terms of tooling, things have only gotten better. You should configure your editor to invoke gofmt on save. (At this point, I hope that’s not in any way controversial.) To the best of my knowledge, the go vet tool produces no false positives, so it’s a good idea to make it part of your precommit hook. And check out the excellent gometalinter for linting concerns. This can produce false positives, so it’s not a bad idea to encode your own conventions somehow.

Configuration original

Configuration is the surface area between the runtime environment and the process. It should be explicit and well-documented. I still use and recommend package flag, but I admit at this point I wish it were less esoteric. I wish it had standard, getopts-style long- and short-form argument syntax, and I wish its usage text were much more compact.

12-factor apps encourage you to use environment vars for configuration, and I think that’s fine, provided each var is also defined as a flag . Explicitness is important: changing the runtime behavior of an application should happen in ways that are discoverable and documented.

I said it in 2014 but I think it’s important enough to say again: define and parse your flags in func main . Only func main has the right to decide the flags that will be available to the user. If your library code wants to parameterize its behavior, those parameters should be part of type constructors. Moving configuration to package globals has the illusion of convenience, but it’s a false economy: doing so breaks code modularity, makes it more difficult for developers or future maintainers to understand dependency relationships, and makes writing independent, parallelizable tests much more difficult.

Top Tip— Only func main has the right to decide which flags are available to the user.

I think there’s a great opportunity for a well-scoped flags package to emerge from the community, combining all of these characteristics. Maybe it already exists; if so, please let me know . I’d certainly use it.

In the talk, I used configuration as a jumping-off point, to discuss a few other issues of program design. (I didn’t cover this in the 2014 version.) To start, let’s take a look at constructors. If we are properly parameterizing all of our dependencies, our constructors can get quite large.

foo, err := newFoo(     *fooKey,     bar,     100 * time.Millisecond,     nil, ) if err != nil {     log.Fatal(err) } defer foo.close() 

Sometimes this kind of construction is best expressed with a config object: a struct parameter to a constructor that takes optional parameters to the constructed object. Let’s assume fooKey is a required parameter, and everything else either has a sensible default or is optional. Often, I see projects construct config objects in a sort of piecemeal way.

// Don't do this. cfg := fooConfig{} cfg.Bar = bar cfg.Period = 100 * time.Millisecond cfg.Output = nil  foo, err := newFoo(*fooKey, cfg) if err != nil {     log.Fatal(err) } defer foo.close() 

But it’s considerably nicer to leverage so-called struct initialization syntax to construct the object all at once, in a single statement.

// This is better. cfg := fooConfig{     Bar:    bar,     Period: 100 * time.Millisecond,     Output: nil, }  foo, err := newFoo(*fooKey, cfg) if err != nil {     log.Fatal(err) } defer foo.close() 

No statements go by where the object is in an intermediate, invalid state. And all of the fields are nicely delimited and indented, mirroring the fooConfig definition.

Notice we construct and then immediately use the cfg object. In this case we can save another degree of intermediate state, and another line of code, by inlining the struct declaration into the newFoo constructor directly.

// This is even better. foo, err := newFoo(*fooKey, fooConfig{     Bar:    bar,     Period: 100 * time.Millisecond,     Output: nil, }) if err != nil {     log.Fatal(err) } defer foo.close() 

Nice.

Top Tip— Use struct literal initialization to avoid invalid intermediate state. Inline struct declarations where possible.

Let’s turn to the subject of sensible defaults. Observe that the Output parameter is something that can take a nil value. For the sake of argument, assume it’s an io.Writer. If we don’t do anything special, when we want to use it in our foo object, we’ll have to first perform a nil check.

func (f *foo) process() {     if f.Output != nil {         fmt.Fprintf(f.Output, "start/n")     }     // ... } 

That’s not great. It’s much safer, and nicer, to be able to use output without having to check it for existence.

func (f *foo) process() {      fmt.Fprintf(f.Output, "start/n")      // ... } 

So we should provide a usable default here. With interface types, one good way is to pass something that provides a no-op implementation of the interface. And it turns out that the stdlib ioutil package comes with a no-op io.Writer, called ioutil.Discard.

Top Tip— Avoid nil checks via default no-op implementations.

We could pass that into the fooConfig object, but that’s still fragile. If the caller forgets to do it at the callsite, we’ll still end up with a nil parameter. So, instead, we can create a sort of safety within the constructor.

func newFoo(..., cfg fooConfig) *foo {     if cfg.Output == nil {         cfg.Output = ioutil.Discard     }     // ... } 

This is just an application of the Go idiom make the zero value useful . We allow the zero value of the parameter (nil) to yield good default behavior (no-op).

Top Tip— Make the zero value useful, especially in config objects.

Let’s revisit the constructor. The parameters fooKey, bar, period, output are all dependencies . The foo object depends on each of them in order to start and run successfully. If there’s a single lesson I’ve learned from writing Go code in the wild and observing large Go projects on a daily basis for the past six years, it is this: make dependencies explicit .

Top Tip— Make dependencies explicit!

An incredible amount of maintenance burden, confusion, bugs, and unpaid technical debt can, I believe, be traced back to ambiguous or implicit dependencies. Consider this method on the type foo.

func (f *foo) process() {     fmt.Fprintf(f.Output, "start/n")     result := f.Bar.compute()     log.Printf("bar: %v", result) // Whoops!     // ... } 

fmt.Printf is self-contained and doesn’t affect or depend on global state; in functional terms, it has something like referential transparency . So it is not a dependency. Obviously, f.Bar is a dependency. And, interestingly, log.Printf acts on a package-global logger object, it’s just obscured behind the free function Printf. So it, too, is a dependency.

What do we do with dependencies? We make them explicit. Because the process method prints to a log as part of its work, either the method or the foo object itself needs to take a logger object as a dependency. For example, log.Printf should become f.Logger.Printf.

func (f *foo) process() {     fmt.Fprintf(f.Output, "start/n")     result := f.Bar.compute()     f.Logger.Printf("bar: %v", result) // Better.     // ... } 

We’re conditioned to think of certain classes of work, like writing to a log, as incidental. So we’re happy to leverage helpers, like package-global loggers, to reduce the apparent burden. But logging, like instrumentation, is often crucial to the operation of a service. And hiding dependencies in the global scope can and does come back to bite us, whether it’s something as seemingly benign as a logger, or perhaps another, more important, domain-specific component that we haven’t bothered to parameterize. Save yourself the future pain by being strict: make all your dependencies explicit.

Top Tip— Loggers are dependencies, just like references to other components, database handles, commandline flags, etc.

Of course, we should also be sure to take a sensible default for our logger.

func newFoo(..., cfg fooConfig) *foo {     // ...     if cfg.Logger == nil {         cfg.Logger = log.NewLogger(ioutil.Discard, ...)     }     // ... } 

Logging and instrumentation original

To speak about the problem generally for a moment: I’ve had a lot more production experience with logging, which has mostly just increased my respect for the problem. Logging is expensive, more expensive than you think, and can quickly become the bottleneck of your system. I wrote more extensively on the subject in a separate blog post , but to re-cap:

  • Log only actionable information , which will be read by a human or a machine
  • Avoid fine-grained log levels — info and debug are probably enough
  • Use structured logging — I’m biased, but I recommend go-kit/log
  • Loggers are dependencies!

Where logging is expensive, instrumentation is cheap. You should be instrumenting every significant component of your codebase. If it’s a resource, like a queue, instrument it according to Brendan Gregg’s USE method : utilization, saturation, and error count (rate). If it’s something like an endpoint, instrument it according to Tom Wilkie’s RED method : request count (rate), error count (rate), and duration.

If you have any choice in the matter, Prometheus is probably the instrumentation system you should be using. And, of course, metrics are dependencies, too!

Let’s use loggers and metrics to pivot and address global state more directly. Here are some facts about Go:

  • log.Print uses a fixed, global log.Logger
  • http.Get uses a fixed, global http.Client
  • database/sql uses a fixed, global driver reigstry
  • func init exists only to have side effects on package-global state

These facts are convenient in the small, but awkward in the large. That is, how can we test the log output of components that use the fixed global logger? We must redirect its output, but then how can we test in parallel? Just don’t? That seems unsatisfactory. Or, if we have two independent components both making HTTP requests with different requirements, how do we manage that? With the default global http.Client, it’s quite difficult. Consider this example.

func foo() {     resp, err := http.Get("http://zombo.com")     // ... } 

http.Get calls on a global in package http. It has an implicit global dependency. Which we can eliminate pretty easily.

func foo(client *http.Client) {     resp, err := client.Get("http://zombo.com")     // ... } 

Just pass an http.Client as a parameter. But that is a concrete type, which means if we want to test this function we also need to provide a concrete http.Client, which likely forces us to do actual HTTP communication. Not great. We can do one better, by passing an interface which can Do (execute) HTTP requests.

type Doer interface {     Do(*http.Request) (*http.Response, error) }  func foo(d Doer) {     req, _ := http.NewRequest("GET", "http://zombo.com", nil)     resp, err := d.Do(req)     // ... } 

http.Client satisfies our Doer interface automatically, but now we have the freedom to pass a mock Doer implementation in our test. And that’s great: a unit test for func foo is meant to test only the behavior of foo, it can safely assume that the http.Client is going to work as advertised.

Speaking of testing…

Testing original

In 2014, I reflected on our experience with various testing frameworks and helper libraries, and concluded that we never found a great deal of utility in any of them, recommending the stdlib’s approach of plain package testing with table-based tests. Broadly, I still think this is the best advice. The important thing to remember about testing in Go is that it is just programming . It is not sufficiently different from other programming that it warrants its own metalanguage. And so package testing continues to be well-suited to the task.

TDD/BDD packages bring new, unfamiliar DSLs and control structures, increasing the cognitive burden on you and your future maintainers. I haven’t personally seen a codebase where that cost has paid off in benefits. Like global state, I believe these packages represent a false economy, and more often than not are the product of cargo-culting behaviors from other languages and ecosystems. When in Go, do as Gophers do : we already have a language for writing simple, expressive tests—it’s called Go, and you probably know it pretty well.

With that said, I do recognize my own context and biases. Like with my opinions on the GOPATH, I’ve softened a bit, and defer to those teams and organizations for whom a testing DSL or framework may make sense. If you know you want to use a package, go for it. Just be sure you’re doing it for well-defined reasons.

Another incredibly interesting topic has been designing for testing. Mitchell Hashimoto recently gave a great talk on the subject here in Berlin ( SpeakerDeck , YouTube ) which I think should be be required viewing.

In general, the thing that seems to work the best is to write Go in a generally functional style, where dependencies are explicitly enumerated, and provided as small, tightly-scoped interfaces whenever possible. Beyond being good software engineering discipline in itself, it feels like it automatically optimizes your code for easy testing.

Top Tip— Use many small interfaces to model dependencies.

As in the http.Client example just above, remember that unit tests should be written to test the thing being tested, and nothing more. If you’re testing a process function, there’s no reason to also test the HTTP transport the request came in on, or the path on disk the results get written to. Provide inputs and outputs as fake implementations of interface parameters, and focus on the business logic of the method or component exclusively.

Top Tip— Tests only need to test the thing being tested.

Dependency management original

Ever the hot topic. In 2014, things were nascent, and about the only concrete advice I could give was to vendor. That advice still holds today: vendoring is still the solution to dependency management for binaries. In particular, the GO15VENDOREXPERIMENT and its concomittant vendor/ subdirectory have become default in Go 1.6. So you’ll be using that layout. And, thankfully, the tools have gotten a lot better. Some I can recommend:

  • FiloSottile/gvt takes a minimal approach, basically just extracting the vendor subcommand from the gb tool so it can be used standalone.
  • Masterminds/glide takes a maximal approach, attempting to recreate the feel and finish of a fully-featured dependency management tool using vendoring under the hood.
  • kardianos/govendor sits in the middle, providing probably the richest interface to vendoring-specific nouns and verbs, and is driving the conversation on the manifest file.
  • constabulary/gb abandons the go tooling altogether in favor of a different repository layout and build mechanism. Great if you produce binaries and can mandate the build environment, e.g. in a corporate setting.

Top Tip— Use a top tool to vendor dependencies for your binary.

A big caveat for libraries. In Go, dependency management is a concern of the binary author. Libraries with vendored dependencies are very difficult to use; so difficult that it is probably better said that they are impossible to use. There are many corner cases and edge conditions that have played out in the months since vendoring was officially introduced in 1.5. (You can dig in to one of these forum posts if you’re particularly interested in the details.) Without getting too deep in the weeds, the lesson is clear: libraries should never vendor dependencies.

Top Tip— Libraries should never vendor their dependencies.

You can carve out an exception for yourself if your library has hermetically sealed its dependencies, so that none of them escape to the exported (public) API layer. No dependent types referenced in any exported functions, method signatures, structures—anything.

If you have the common task of maintaining an open-source repository that contains both binaries and libraries, unfortunately, you are stuck between a rock and a hard place. You want to vendor your deps for your binaries, but you shouldn’t vendor them for your libraries, and the GO15VENDOREXPERIMENT doesn’t admit this level of granularity, from what appears to me to be regrettable oversight.

Bluntly, I don’t have an answer to this. The etcd folks have hacked together a solution using symlinks which I cannot in good faith recommend, as symlinks are not well-supported by the go toolchain and break entirely on Windows. That this works at all is more a happy accident than any consequence of design. I and others have raised all of these concerns to the core team , and I hope something will happen in the near term.

Build and deploy original

Regarding building, one important lesson learned, with a hat tip to Dave Cheney: prefer go install to go build. The install verb caches build artifacts from dependencies in $GOPATH/pkg, making builds faster. It also puts binaries in $GOPATH/bin, making them easier to find and invoke.

Top Tip— Prefer go install to go build.

If you produce a binary, don’t be afraid to try out new build tools like gb , which may significantly reduce your cognitive burden. Conversely, remember that since Go 1.5 cross-compilation is built-in; just set the appropriate GOOS and GOARCH environment variables, and invoke the appropriate go command. So there’s no need for extra tools here anymore.

Regarding deployment, we Gophers have it pretty easy compared to languages like Ruby or Python, or even the JVM. One note: if you deploy in containers, follow the advice of Kelsey Hightower and do it FROM scratch. Go gives us this incredible opportunity; it’s a shame not to use it.

As more general advice, think carefully before choosing a platform or orchestration system—if you even choose one at all. Likewise for jumping onto the microservices bandwagon. An elegant monolith, deployed as an AMI to an autoscaling EC2 group, is a very productive setup for small teams. Resist, or at least carefully consider, the hype.

The Top Tips:

  1. Put $GOPATH/bin in your $PATH, so installed binaries are easily accessible.  
  2. If your repo foo is primarily a binary, put your library code in a lib/ subdir, and call it package foo.  
  3. If your repo is primarily a library, put your binaries in separate subdirectories under cmd/.  
  4. Defer to Andrew Gerrand’s naming conventions .  
  5. Only func main has the right to decide which flags are available to the user.  
  6. Use struct literal initialization to avoid invalid intermediate state.  
  7. Avoid nil checks via default no-op implementations.  
  8. Make the zero value useful, especially in config objects.  
  9. Make dependencies explicit!   
  10. Loggers are dependencies, just like references to other components, database handles, commandline flags, etc.  
  11. Use many small interfaces to model dependencies.  
  12. Tests only need to test the thing being tested.  
  13. Use a top tool to vendor dependencies for your binary.  
  14. Libraries should never vendor their dependencies.  
  15. Prefer go install to go build.  

Go has always been a conservative language, and its maturity has brought relatively few surprises and effectively no major changes. Consequently, and predictably, the community also hasn’t dramatically shifted its stances on what’s considered best practice. Instead, we’ve seen a reification of tropes and proverbs that were reasonably well-known in the early years, and a gradual movement “up the stack” as design patterns, libraries, and program structures are explored and transformed into idiomatic Go.

Here’s to another 6 years of fun and productive Go programming. ��

转载本站任何文章请注明:转载至神刀安全网,谢谢神刀安全网 » Go best practices, six years in

分享到:更多 ()

评论 抢沙发

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址