cos

Concurrency Programming#

• Concurrency is running multi-threaded programs on a single-core CPU
  

• Parallelism is running multi-threaded programs on multiple cores
  

• Go can fully leverage multi-core advantages and run efficiently
  An important concept

Goroutines#

• Goroutines have less overhead than threads and can be understood as lightweight threads. A Go program can create tens of thousands of goroutines.

Starting a goroutine in Go is very simple; just add the go keyword before a function to start a goroutine for that function.

CSP and Channel#

CSP (Communicating Sequential Process)

Go advocates for communicating by sharing memory rather than sharing memory to communicate.

So how do we communicate? Through channels.

Channel#

Syntax: make(chan element type, [buffer size])

• Unbuffered channel make(chan int)
• Buffered channel make(chan int, 2)
  This diagram is very vivid and illustrative~
  

Here is an example:

• The first goroutine acts as a producer sending 0~9 to src.
• The second goroutine acts as a consumer calculating the square of each number in src and sending it to dest.
• The main thread outputs each number in dest.

package main

func CalSquare() {
   src := make(chan int)     // Producer
   dest := make(chan int, 3) // Consumer with buffer to solve the problem of the producer being too fast
   go func() {               // This thread sends 0~9 to src
      defer close(src) // defer means to execute at the end of the function to release allocated resources.
      for i := 0; i < 10; i++ {
         // <- operator left side is the collector of data, right side is the data to be sent
         src <- i
      }
   }() // Immediately executed
   go func() {
      defer close(dest)
      for i := range src {
         dest <- i * i
      }
   }()
   for i := range dest {
      // Other complex operations
      println(i)
   }
}
func main() {
   CalSquare()
}

As we can see, the output is always sequential, indicating that Go is concurrently safe.

The Go language also retains the practice of shared memory, using sync for synchronization, as shown below.

package main

import (
   "sync"
   "time"
)

var (
   x    int64
   lock sync.Mutex
)

func addWithLock() { // x adds up to 2000, using a lock is very safe
   for i := 0; i < 2000; i++ {
      lock.Lock() // Lock
      x += 3
      x -= 2
      lock.Unlock() // Unlock
   }
}
func addWithoutLock() { // Without using a lock
   for i := 0; i < 2000; i++ {
      x += 3
      x -= 2
   }
}
func Add() {
   x = 0
   for i := 0; i < 5; i++ {
      go addWithoutLock()
   }
   time.Sleep(time.Second) // Sleep for 1s
   println("WithoutLock x =", x)
   x = 0
   for i := 0; i < 5; i++ {
      go addWithLock()
   }
   time.Sleep(time.Second) // Sleep for 1s
   println("WithLock x =", x)
}
func main() {
   Add()
}

ps: Tried many times without conflict, fun. Changing the calculations slightly makes conflicts arise.

Dependency Management#

No large project development can avoid dependency management. Dependencies in Go have mainly evolved from GOPATH -> Go Vendor -> Go Module, and now the main method is to use Go Module.

• Different environments have different versions of dependencies, so how do we control the versions of dependency libraries?

GOPATH#

• Project code directly depends on the code under src.
• Use go get to download the latest version of the package to the src directory.

This leads to a problem: it cannot achieve multi-version control (A and B depend on different versions of the same package, sigh).

Go Vendor#

• A vendor folder is added to the project directory, where all dependency package copies are stored.
• By using vendor => GOPATH, we can find a workaround.

ps: It feels quite similar to the front-end package.json... Dependency issues are really unavoidable.

This also creates new problems:

• Cannot control the versions of dependencies.
• Updating the project may lead to dependency conflicts, resulting in compilation errors.

Go Module#

• Manage dependency package versions through the go.mod file.
• Manage dependency packages using go get/go mod command tools.

Achieving the ultimate goal: defining version rules while managing project dependencies.

It can be compared to Maven in Java.

Dependency Configuration go.mod#

Dependency identification syntax: module path + version for unique identification.

[Module Path][Version/Pseudo-version]

module example/project/app     Basic unit of dependency management

go 1.16     Native library

require (    Unit dependencies
    example/lib1 v1.0.2
    example/lib2 v1.0.0 // indirect
    example/lib3 v0.1.0-20190725025543-5a5fe074e612
    example/lib4 v0.0.0-20180306012644-bacd9c7ef1dd // indirect
    example/lib5/v3 v3.0.2
    example/lib6 v3.2.0+incompatible
)

As above, it should be noted that:

• Major version 2+ modules will have a /vN suffix added to the path.
• For dependencies without a go.mod file and major version 2+, it will be +incompatible.
  The version rules for dependencies are divided into semantic versions and commit-based pseudo-versions.

Semantic Versioning#

The format is: ${MAJOR}.${MINOR}.${PATCH} V1.3.0, V2.3.0, ……

• Different MAJOR versions indicate incompatible APIs.
  • Even if it is the same library, different MAJOR versions will be considered different modules.
• MINOR versions usually indicate new functions or features, backward compatible.
• PATCH versions generally indicate bug fixes.

Commit-based Versions#

The format is: ${vx.0.0-yyyymmddhhmmss-abcdefgh1234}

• The version prefix is the same as semantic versions.
• Timestamp (yyyymmddhhmmss), which is the commit time.
• Checksum (abcdefgh1234), a 12-digit hash prefix.
  • After each commit, Go will automatically generate a pseudo-version number.

Small Test#

1. If project X depends on projects A and B, and A and B depend on versions v1.3 and v1.4 of project C respectively, as shown in the dependency graph, what version of project C will be used during final compilation? []{.gap} ？ {.quiz}
   • v1.3
   • v1.4 {.correct}
   • A compiles with v1.3 when using C, and B compiles with v1.4 when using C.
     {.options}

   The answer is: B chooses the lowest compatible version
   This is the algorithm Go uses for version selection, choosing the lowest compatible version, and version 1.4 is backward compatible with 1.3 (semantic versioning). Why not choose 1.3? Because it does not provide upward compatibility, if there is also a 1.5, it would not choose 1.5, because 1.4 meets the requirements for the lowest compatible version.

Dependency Distribution#

Where do these dependencies get downloaded from? That is dependency distribution.

Download from corresponding repositories on code hosting systems like GitHub?

GitHub is a common code hosting platform, and dependencies defined in the Go Modules system can ultimately correspond to a specific commit or version of a project in a multi-version code management system.

For dependencies defined in go.mod, they can be downloaded from the corresponding repository to complete dependency distribution.

There are also issues:

• Cannot guarantee build determinism.
  • Software authors directly modify software versions, leading to the next build using different versions of dependencies or being unable to find dependency versions.
• Cannot guarantee dependency availability.
  • Software authors directly delete software from code platforms, leading to unavailable dependencies.
• Increases pressure on third-party code hosting platforms.

These issues can be resolved through the Proxy method.

Go Proxy is a service site that caches software content from the source site, and the cached software versions do not change, and they remain available even after the source site software is deleted.

After using Go Proxy, dependencies will be pulled directly from the Go Proxy site during the build.

Go Modules control how to use Go Proxy through the GOPROXY environment variable.

Service site URL list, direct indicates the source site: GOPROXY="https://proxy1.cn, https://proxy2.cn,direct"

• GOPROXY is a list of Go Proxy site URLs, and direct can be used to indicate the source site. The overall dependency addressing path will prioritize downloading dependencies from proxy1, if proxy1 does not exist, it will look for them in proxy2, and if proxy2 also does not exist, it will fallback to the source site to download dependencies directly, caching them in the proxy site.

Tools#

go get example.org/pkg

go mod

Testing#

Testing is generally divided into regression testing, integration testing, and unit testing, with coverage increasing layer by layer from front to back, while costs decrease layer by layer, so the coverage of unit testing to some extent determines the quality of the code.

• Regression testing is generally done manually by QA colleagues through terminals to regress some fixed main process scenarios.
• Integration testing verifies the system's functional dimensions.
• Unit testing is done during the development phase, where developers verify the functionality of individual functions and modules.

Unit testing mainly includes: input, test unit, output, and verification.

The concept of a unit is broad, including interfaces, functions, modules, etc., with the final verification ensuring that the code's functionality matches our expectations.

Unit testing has the following benefits:

• Ensures quality
  • When overall coverage is sufficient, it ensures the correctness of new features while not breaking the correctness of existing code.
• Increases efficiency
  • In the case of code bugs, unit tests can help locate and fix issues within a short cycle.

Go has the following rules for unit testing:

• All test files end with _test.go
• func TestXxx(testing.T)
• Initialization logic is placed in the TestMain function (data loading configuration before testing, resource release after testing, etc.).

Example:
main.go

package main

func HelloTom() string {
   return "Jerry"
}

main_test.go

package main

import "testing"

func TestHelloTom(t *testing.T) {
   output := HelloTom()
   expectOutput := "Tom"
   if output != expectOutput {
      t.Errorf("Expect %s do not match actual %s", expectOutput, output)
   }
}

In actual projects, unit test coverage

• The general requirement for projects is 50%~60% coverage.
• For important financial services, coverage may be required to reach 80%.

Unit tests need to ensure stability and idempotency.

• Stability means mutual isolation, able to run tests at any time and in any environment.
• Idempotency means that each test run should produce the same result as before.

To achieve this goal, the mock mechanism is used.

bouk/monkey: Monkey patching in Go

Monkey is an open-source mock testing library that can mock methods or instance methods, using reflection and pointer assignment. The scope of Mockey Patch is at Runtime, allowing the address of function A in memory to be replaced with the address of function B at runtime, redirecting the implementation of the function to be stubbed.

Go also provides a benchmarking testing framework.

• Benchmark testing refers to testing the performance of a piece of code and the extent of CPU consumption.

In actual project development, we often encounter performance bottlenecks in code, and to locate issues, we frequently need to conduct performance analysis, which is where benchmark testing comes into play. The usage is similar to unit testing.

Mentioned fastrand, address: bytedance/gopkg: Universal Utilities for Go

Summary and Insights#

This lesson mainly covered concurrency management, dependency configuration, and testing in Go. There is a lot of content that needs to be digested. There will also be a project practice session later, which will be conducted tomorrow.

The content of this lesson is derived from the course by Teacher Zhao Zheng of the third Youth Training Camp.