Knowledgebase

How to Build a Web API with gRPC and Protocol Buffers in Go Print

  • 0

Introduction

This article describes gRPC, protocol buffers, and how to use these tools to build a simple web API in Go.

Prerequisites

  • Working knowledge of Go.
  • Properly installed Go toolchain (Go 1.x installed).
  • Knowledge of how APIs work.

gRPC

gRPC (gRPC Remote Procedure Calls) is a modern, open-source, high-performance RPC framework that can run in any environment and uses HTTP 2.0 as the underlying transport protocol. It enables transparent communication between clients and servers and simplifies the building of connected systems.

During server-client communication in gRPC, as in many other RPC systems, a client application can directly invoke a method on another system - as if it was a local call, this makes it easier to build highly efficient distributed services. gRPC clients and servers can run in a variety of environments and be implemented in various supported languages; you can build a gRPC server in Go and implement clients in Ruby, Python, or any one of the other supported languages.

In gRPC, you define a service to specify what methods can be called remotely with the corresponding parameters and return types. On the server-side, this interface is implemented, and a gRPC server is run to handle client calls, while on the client-side, a client that provides access to the same methods as a server - is run.

As we mentioned earlier, gRPC utilizes HTTP 2.0 as the underlying transport protocol. This allows gRPC to get the most out of HTTP 2 advantages, including streaming communication and bidirectional support. Typically, in the request-response communication model built-in HTTP 1.1 used by REST APIs, if a service receives multiple requests from multiple clients - the service has to respond to each request one at a time; this, in turn, slows down the entire system, but in HTTP 2.0's client-response communication model that supports streaming and includes bidirectional support, gRPC can receive multiple requests from multiple clients, and handle those requests simultaneously by constantly streaming information.

gRPC supports the following types of streams:

  • Unary streams - a single request-response pair between client and server.
  • Server streams - the server responds with a stream of messages to a client's request.
  • Client streams - the client sends a stream of messages to the server, and the server, in turn, responds with a single response.
  • Bidirectional streams - both streams (client and server) are independent of each other and can both transmit messages in any order. The client is the one who initiates and ends a bidirectional stream.

By default, gRPC utilizes a binary format called Protocol Buffers to serialize data; gRPC offers very high performance and is lightweight.

Protocol Buffers

Protocol Buffers binary format (Protobuf) is an open-source, cross-platform data format used to serialize structured data. You define the structure of the data you want to serialize in a proto file, a normal text file with a .proto extension. Then, use the protocol buffer compiler, protoc - to generate data access classes in your preferred language using your proto definition.

Protocol buffer data is structured as messages, with name-value pairs:

message Student {
  string name = 1;
  int32 id = 2;
  bool active = 3;
}

These messages are used as parameters and return types for gRPC services defined in proto files, with RPC methods:

// The Echo service definition
service Echo {
// Sends a greeting back
rpc SayHi (HiRequest) returns (HiResponse) {}
}

// The request message containing the user’s name
message HiRequest {
  string name = 1;
}

// The response message containing a greeting
message HiReply {
  string message = 1;
}

In gRPC, you create a service that has RPC methods; these methods take a request message and return a response message.

gRPC uses protoc with a special plugin to generate code from your proto file; it generates the gRPC server and client code, as well as the regular protocol buffer code for serialiing and retrieving your message types.

Now that we've covered what gRPC and protocol buffers entail, let's get started building our gRPC web API by defining our API resources and installing the required tools.

Identifying our API resources

The first step to building APIs is to define what endpoints will be available via your API. We will be building a simple gRPC web API that creates or reads a user resource from our server; we have the following endpoints:

  • API Method FetchUser: Fetch a user
  • API Method CreateUser: Creates a user
  • API Method UpdateUser: Updates a user
  • API Method DeleteUser: Deletes a user

Let's get started with development - by installing the required libraries (gRPC and protocol buffers).

Installing gRPC

To install gRPC, run the following command in your terminal:

$ go get -u google.golang.org/grpc

This will install the gRPC packages to your Go toolchain.

Installing Protocol Buffers

Next, let's install the protobuf compiler:

$ sudo apt install -y protobuf-compiler

Running the above command will install the protocol buffer compiler, which we will be using to generate code from our proto file, later on, using the protoc command.

Creating our project

Let's create our project in our GOPATH:

$ mkdir $GOPATH/src/grpc-api

Now cd into our project directory:

$ cd $GOPATH/src/grpc-api

Defining our proto file

Now create a new folder for our proto files:

$ mkdir protobuf

Let’s create our proto file:

$ touch protobuf/protobuf.proto

Open our newly created file in your text editor, and add the following lines:

syntax = "proto3";

package protobuf;

The first line tells the protocol buffer compiler that we're using proto3 syntax, or else the compiler will assume that we are using proto2. This must be the first line non-empty, non-comment line in a proto file. Then, we specify the package this file belongs to.

Next, we define our message and its field types:

message User {
  int32 uid = 1;
  string name = 2;
  string nationality = 3;
  int32 zip = 4;
}

This specifies a User message type that takes four fields consisting of two integers (uid and zip), and two strings (name and nationality). You can also use other messages as field types. The number assigned to each field in the message definition is uniquely used to identify the field in the message binary format and should not be changed once the message type is in use.

Next, let’s define our message request/response pair for our FetchUser method:

message FetchUserRequest {
  int32 uid = 1;
}

message FetchUserResponse {
  User user = 1;
}

In our FetchUserRequest message, we accept a user ID (uid) that will be used to find the requested user. We'll return the requested user in a FetchUserResponse message that takes a User message type that we defined earlier as a field.

Then, we define similar request/response message pairs for the rest of our methods:

message CreateUserRequest {
  User user = 1;
}

message CreateUserResponse {
  User user = 1;
}

message UpdateUserRequest {
  User user = 1;
}

message UpdateUserResponse {
  User user = 1;
}

message DeleteUserRequest {
  int32 uid = 1;
}

message DeleteUserResponse {
  int32 uid = 1;
}

Above, we define CreateUserRequest and CreateUserResponse, which both take a User message type as fields. So, when a client sends a User in the CreateUserRequest, we send back the created user in a CreateUserResponse message. We also create request/response pairs for the rest of our methods, UpdateUser and DeleteUser.

Now that we’ve defined our message request/response pairs, let’s create our service and register our RPC methods (FetchUser, CreateUser, UpdateUser, and DeleteUser) under the service:

service UserService {
  rpc FetchUser(FetchUserRequest) returns (FetchUserResponse);
  rpc CreateUser(CreateUserRequest) returns (CreateUserResponse);
  rpc UpdateUser(UpdateUserRequest) returns (UpdateUserResponse);
  rpc DeleteUser(DeleteUserRequest) returns (DeleteUserResponse);
}

Here, we defined a service called UserService, and registered our RPC methods with their parameters and return values, e.g. the FetchUser method accepts a FetchUserRequest as a parameter and returns a FetchUserResponse, etc.

Now our protobuf.proto file should look like this:

syntax = "proto3";

package protobuf;

message User {
  int32 uid = 1;
  string name = 2;
  string nationality = 3;
  int32 zip = 4;
}

message FetchUserRequest {
  int32 uid = 1;
}

message FetchUserResponse {
  User user = 1;
}

message CreateUserRequest {
  User user = 1;
}

message CreateUserResponse {
  User user = 1;
}

message UpdateUserRequest {
  User user = 1;
}

message UpdateUserResponse {
  User user = 1;
}

message DeleteUserRequest {
  int32 uid = 1;
}

message DeleteUserResponse {
  int32 uid = 1;
}

service UserService {
  rpc FetchUser(FetchUserRequest) returns (FetchUserResponse);
  rpc CreateUser(CreateUserRequest) returns (CreateUserResponse);
  rpc UpdateUser(UpdateUserRequest) returns (UpdateUserResponse);
  rpc DeleteUser(DeleteUserRequest) returns (DeleteUserResponse);
}

To compile our proto file with the protocol buffer compiler and generate the corresponding Go code to use, simply run the command from our project directory:

$ protoc --gogo_out=plugins=grpc:. protobuf/protobuf.proto

Running this command will generate a .pb.go file in our protobuf directory, which contains the equivalent Go bindings from our proto file which implements the gRPC code our application will use, along with server and client methods.

Creating our Server

We can build our server after generating Go code from our protocol buffer compiler (using protoc).

Let's create our server.go file:

$ touch server.go

Open the file in your favorite text editor, and add the following lines:

package main

import (
  "context"
  "fmt"
  "log"
  "net"
  "errors"
  "time"
  "math"

  "google.golang.org/grpc"
  "google.golang.org/grpc/keepalive"

  "grpc-api/protobuf"
)

We've just imported the required libraries, including our protobuf package, so that we can access the methods and structs generated by protoc.

The protocol buffer compiler generated a UserServiceServer interface for our UserService, which implements the RPC methods that we defined.

Before we go on to define our server, let's create a struct that we can use as our database:

type userDetails struct {
  Uid int32
  Name string
  Nationality string
  Zip int32
}

var users = []userDetails{
  {
    Uid: 1,
    Name: "Josh Winters",
    Nationality: "American",
    Zip: 10111,
  },
  {
    Uid: 2,
    Name: "Brian Stone",
    Nationality: "British",
    Zip: 20212,
  },
}

We've created a userDetails struct with identical fields to the User type defined in our proto file; we will be using a slice of userDetails as our database for reading, updating, creating, and deleting users. Let's create helper functions to help convert between User and userDetails type:

func toUser(data userDetails) *protobuf.User {
  return &protobuf.User {
    Uid: data.Uid,
    Name: data.Name,
    Nationality: data.Nationality,
    Zip: data.Zip,
  }
}

func fromUser(user *protobuf.User) userDetails {
  return userDetails {
    Uid: user.GetUid(),
    Name: user.GetName(),
    Nationality: user.GetNationality(),
    Zip: user.GetZip(),
  }
}

Now we can convert back and forth from User to userDetails easily, with our helper functions; we use the Get<fieldname> method to extract the values from our User type.

Next, we define a server struct that will satisfy the UserServiceServer interface generated by protoc:

type server struct {
  protobuf.UserServiceServer
}

Now, let’s define our methods for the server:

FetchUser() method

func (s *server) FetchUser(ctx context.Context, req *protobuf.FetchUserRequest) (*protobuf.FetchUserResponse, error) {
  fmt.Println("Fetching User")
  uid := req.GetUid()

  for _, user := range users {
    if user.Uid == uid {
    return &protobuf.FetchUserResponse {
        User: toUser(user),
      }, nil
    }
  }

  return nil, errors.New("User not found")
}

The generated FetchUser() method accepts two arguments; a context and a FetchUserRequest and returns a FetchUserResponse and an error - so to implement it we pass it a context and FetchUserRequest as arguments.

In our implementation, we print a "Fetching User" message to the screen and use the GetUid() method generated by protoc in the .pb.go file to get the Uid from the FetchUserRequest. Then, we iterate through our list of users and return the user if found else we return nil and an error.

CreateUser() method

func (s *server) CreateUser(ctx context.Context, req *protobuf.CreateUserRequest) (*protobuf.CreateUserResponse, error) {
  fmt.Println("Creating User")
  user := req.GetUser()

  data := fromUser(user)

  users = append(users, data)
  
  return &protobuf.CreateUserResponse {
    User: toUser(data),
  }, nil
}

Here, we read the user from the CreateUserRequest using the GetUser() method generated by the compiler - to get the User from the request; next, we convert the user to a userDetails type using our fromUser helper function, then append the user to our list of users and send back the created user as a response.

UpdateUser() method

func (s *server) UpdateUser(ctx context.Context, req *protobuf.UpdateUserRequest) (*protobuf.UpdateUserResponse, error) {
  fmt.Println("Updating User")
  user := req.GetUser()

  data := fromUser(user)

  for i, user := range users {
    if user.Uid == data.Uid {
      users[i] = data
      return &protobuf.UpdateUserResponse {
        User: toUser(data),
      }, nil
    }
  }

  return nil, errors.New("Couldn't update user")
}

Next, we implement our UpdateUser method; first, we get the user from the UpdateUserRequest by using the GetUser() method, and iterate through our list of users searching for the corresponding Uid, if found - we update the user, and return the updated user as a response, else we return nil and an error.

DeleteUser() method

func (s *server) DeleteUser(ctx context.Context, req *protobuf.DeleteUserRequest) (*protobuf.DeleteUserResponse, error) {
  fmt.Println("Deleting User")
  uid := req.GetUid()
  var tmpUsers []userDetails

  for i, user := range users {
    if user.Uid == uid {
      tmpUsers = append(users[:i], users[i+1:]...)
      users = tmpUsers
      fmt.Printf("User with id %d has been deleted.\n", uid)
      return &protobuf.DeleteUserResponse {
        Uid: uid,
      }, nil
    }
  }

  return nil, errors.New("User does not exist")
}

We implement the DeleteUser() method, by fetching the Uid() from the request, and iterating through our list of users, looking for the matching Uid; if found, we delete the user from the list and return the deleted Uid, else we return nil and error if the corresponding user with the Uid was not found.

Creating our main function

Now that we have implemented our server struct which satisfies the interface by implementing all our methods, we can proceed with creating the main code to host the server:

func main() {
  lis, err := net.Listen("tcp", "127.0.0.1:5000")
  if err != nil {
    log.Fatalf("Failed to listen: %v", err)
  }

  opts := []grpc.ServerOption{
    grpc.MaxRecvMsgSize(math.MaxInt64),
    grpc.KeepaliveParams(
      keepalive.ServerParameters{
        Timeout: 5 * time.Second,
      },
    ),
  }

  s := grpc.NewServer(opts...)
  protobuf.RegisterUserServiceServer(s, &server{})

  fmt.Println("Starting server...")
  fmt.Printf("Hosting server on: %s\n", lis.Addr().String())

  if err := s.Serve(lis); err != nil {
    log.Fatalf("Failed to serve: %v", err)
  }
}

First, we create a listener on 127.0.0.1:5000 using:

lis, err := net.Listen(tcp”, 127.0.0.1:5000”)

We handle any errors creating the listener by calling log.Fatalf, which logs the error and calls os.Exit().

Using []grpc.ServerOptions, we create a slice of options for our server and set the maximum receive message size and server timeout.

We then create our gRPC server and pass our slice of ServerOptions as a parameter:

s := grpc.NewServer(opts…)

After creating our gRPC server, we have to register it for our service:

protobuf.RegisterUserServiceServer(s, &server{})

The RegisterUserServiceServer was generated by protoc, and it takes two arguments - a grpc.NewServer() instance, and a server struct that implements all the RPC methods added to our service; we then pass our implemented server struct as the second argument. This binds our gRPC server to our UserService.

Afterward, we host the server by calling s.Serve() with our listener as a parameter, and add proper error handling.

Putting it all together, our server.go script should look like this:

package main

import (
  "context"
  "fmt"
  "log"
  "net"
  "errors"
  "math"
  "time"

  "google.golang.org/grpc"
  "google.golang.org/grpc/keepalive"

  "grpc-api/protobuf"
)

type userDetails struct {
  Uid int32
  Name string
  Nationality string
  Zip int32
}

var users = []userDetails{
  {
    Uid: 1,
    Name: "Josh Winters",
    Nationality: "American",
    Zip: 10111,
  },
  {
    Uid: 2,
    Name: "Brian Stone",
    Nationality: "British",
    Zip: 20212,
  },
}

func toUser(data userDetails) *protobuf.User {
  return &protobuf.User {
    Uid: data.Uid,
    Name: data.Name,
    Nationality: data.Nationality,
    Zip: data.Zip,
  }
}

func fromUser(user *protobuf.User) userDetails {
  return userDetails {
    Uid: user.GetUid(),
    Name: user.GetName(),
    Nationality: user.GetNationality(),
    Zip: user.GetZip(),
  }
}

type server struct {
  protobuf.UserServiceServer
}

func (s *server) FetchUser(ctx context.Context, req *protobuf.FetchUserRequest) (*protobuf.FetchUserResponse, error) {
  fmt.Println("Fetching User")
  uid := req.GetUid()

  for _, user := range users {
    if user.Uid == uid {
      return &protobuf.FetchUserResponse {
        User: toUser(user),
      }, nil
    }
  }

  return nil, errors.New("User not found")
}

func (s *server) CreateUser(ctx context.Context, req *protobuf.CreateUserRequest) (*protobuf.CreateUserResponse, error) {
  fmt.Println("Creating User")
  user := req.GetUser()

  data := fromUser(user)

  users = append(users, data)

  return &protobuf.CreateUserResponse {
    User: toUser(data),
  }, nil
}

func (s *server) UpdateUser(ctx context.Context, req *protobuf.UpdateUserRequest) (*protobuf.UpdateUserResponse, error) {
  fmt.Println("Updating User")
  user := req.GetUser()

  data := fromUser(user)

  for i, user := range users {
    if user.Uid == data.Uid {
      users[i] = data
      return &protobuf.UpdateUserResponse {
        User: toUser(data),
      }, nil
    }
  }

  return nil, errors.New("Couldn't update user")
}

func (s *server) DeleteUser(ctx context.Context, req *protobuf.DeleteUserRequest) (*protobuf.DeleteUserResponse, error) {
  fmt.Println("Deleting User")
  uid := req.GetUid()
  var tmpUsers []userDetails

  for i, user := range users {
    if user.Uid == uid {
      tmpUsers = append(users[:i], users[i+1:]...)
      users = tmpUsers
      fmt.Printf("User with id %d has been deleted.\n", uid)
      return &protobuf.DeleteUserResponse {
        Uid: uid,
      }, nil
    }
  }

  return nil, errors.New("User does not exist")
}

func main() {
  lis, err := net.Listen("tcp", "127.0.0.1:5000")
  if err != nil {
    log.Fatalf("Failed to listen: %v", err)
  }

  opts := []grpc.ServerOption{
    grpc.MaxRecvMsgSize(math.MaxInt64),
    grpc.KeepaliveParams(
      keepalive.ServerParameters{
        Timeout: 5 * time.Second,
      },
    ),
  }

  s := grpc.NewServer(opts...)
  protobuf.RegisterUserServiceServer(s, &server{})

  fmt.Println("Starting server...")
  fmt.Printf("Hosting server on: %s\n", lis.Addr().String())

  if err := s.Serve(lis); err != nil {
    log.Fatalf("Failed to serve: %v", err)
  }
}

Now that we've built our server, let's build our client.

Creating the Client

In an RPC system, client calls are executed on the server like local calls; so the client is simply going to call our methods, and the server will process them.

Let's create our client.go file:

$ touch client.go

Now, open up our newly created file in your favorite text editor, and add the following lines:

package main

import (
  "context"
  "fmt"
  "log"
  
  "google.golang.org/grpc"
  
  "grpc-api/protobuf"
)

Now we’ve imported the required packages, let’s create our main function:

func main() {
  fmt.Println("Starting Client\n")

  cc, err := grpc.Dial("127.0.0.1:5000", grpc.WithInsecure())
  if err != nil {
    log.Fatalf("Error connecting: %v", err)
  }
  // Close connection before exiting application
  defer cc.Close()

  c := protobuf.NewUserServiceClient(cc)

First, we create a connection to our server address at "127.0.0.1:5000" using grpc.Dial(). grpc.Dial() takes transport security as a second parameter. Since we're connecting on the same host machine, we don't need any layer of additional security set, so we simply use grpc.WithInsecure(). Then we perform error handling and defer closing the connection.

Next, we register our connection as a client for our service, using:

c := protobuf.NewUserServiceClient(cc)

To test our CreateUser RPC method, we simply create a new User message, and call the corresponding CreateUser() method:

Calling CreateUser()

// Create a user
fmt.Println("Calling CreateUser()")
user := &protobuf.User {
  Uid: 3,
  Name: "Sarah Connors",
  Nationality: "Canadian",
  Zip: 45015,
}

createUserResponse, err := c.CreateUser(context.Background(), protobuf.CreateUserRequest{User: user})
if err != nil {
  log.Fatalf("An error occurred: %v", err)
}
fmt.Printf("User has been created: %v\n\n", createUserResponse)

We create a new protobuf.User object, and call the CreateUser() method with a context.Background() and our newly created User as a CreateUserRequest. Remember that our CreateUser method takes a CreateUserRequest as an argument, so we pass our declared user as a value for the User field in the CreateUserRequest body.

Then we get back a CreateUserResponse, and print it to the screen.

Now, let's call our other methods to make sure they are all working as expected:

Calling UpdateUser()

// Update a user
updateUser := &protobuf.User {
  Uid: 1,
  Name: "Mandy Williams",
  Nationality: "American",
  Zip: 10111,
}

fmt.Println("Updating a user")
updateUserResponse, err := c.UpdateUser(context.Background(), &protobuf.UpdateUserRequest{User: updateUser})
if err != nil {
  log.Fatalf("Error occurred updating user: %v", err)
}
fmt.Printf("A user has been updated: %v\n\n", updateUserResponse)

Above, we framed a User message with the changes that we want made to that Uid, and called the UpdateUser method. Now the other fields of the User with Uid 1, will be changed on our server.

Calling DeleteUser()

fmt.Println("Deleting user")
var delUid int32 = 2
deleteUserResponse, err := c.DeleteUser(context.Background(), &protobuf.DeleteUserRequest{Uid: delUid})
if err != nil {
  log.Fatalf("Error occurred deleting user: %v", err)
}
fmt.Printf("User with ID: %d has been deleted.\n\n", deleteUserResponse.GetUid())

Here we called DeleteUser() on the User with a Uid of 2.

Calling FetchUser()

// Get a user
fmt.Println("Fetching a user")
var getUid int32 = 1
fetchUserResponse, err := c.FetchUser(context.Background(), &protobuf.FetchUserRequest{Uid: getUid})
if err != nil {
  log.Fatalf("User not found error: %v", getUid)
}
fmt.Printf("User: %v\n", fetchUserResponse)
}

Now, to make sure our update worked, we will call FetchUser on the user with a Uid of 1, to see if the details were changed successfully to our framed request earlier.

Our client.go file should now look like this:

package main

import (
  "context"
  "fmt"
  "log"
  
  "google.golang.org/grpc"

  "grpc-api/protobuf"
)

func main() {
  fmt.Println("Starting Client\n")

  cc, err := grpc.Dial("localhost:5000", grpc.WithInsecure())
  if err != nil {
    log.Fatalf("Error connecting: %v", err)
  }
  // Close connection before exiting app
  defer cc.Close()

  c := protobuf.NewUserServiceClient(cc)

  // Create a user
  fmt.Println("Calling CreateUser()")
  user := &protobuf.User {
    Uid: 3,
    Name: "Sarah Connors",
    Nationality: "Canadian",
    Zip: 45015,
  }

  createUserResponse, err := c.CreateUser(context.Background(), &protobuf.CreateUserRequest{User: user})
  if err != nil {
    log.Fatalf("An error occurred: %v", err)
  }
  fmt.Printf("User has been created: %v\n\n", createUserResponse)

  // userID := createUserResponse.GetUser().GetId()

  // Update a user
  updateUser := &protobuf.User {
    Uid: 1,
    Name: "Mandy Williams",
    Nationality: "American",
    Zip: 10111,
  }

  fmt.Println("Updating a user")
  updateUserResponse, err := c.UpdateUser(context.Background(), &protobuf.UpdateUserRequest{User: updateUser})
  if err != nil {
    log.Fatalf("Error occurred updating user: %v", err)
  }
  fmt.Printf("A user has been updated: %v\n\n", updateUserResponse)

  fmt.Println("Deleting user")
  var delUid int32 = 2
  deleteUserResponse, err := c.DeleteUser(context.Background(), &protobuf.DeleteUserRequest{Uid: delUid})
  if err != nil {
    log.Fatalf("Error occurred deleting user: %v", err)
  }
  fmt.Printf("User with ID: %d has been deleted.\n\n", deleteUserResponse.GetUid())

  // Get a user
  fmt.Println("Fetching a user")
  var getUid int32 = 1
  fetchUserResponse, err := c.FetchUser(context.Background(), &protobuf.FetchUserRequest{Uid: getUid})
  if err != nil {
    log.Fatalf("User not found error: %v", getUid)
  }
  fmt.Printf("User: %v\n", fetchUserResponse)
}

Running the Server and Client

We have successfully built our gRPC server and client, now to test out both of them.

Open up a terminal in our project directory and run the server first:

$ go run server.go

We'll get back our printed message:

Starting server…
Hosting server on 127.0.0.1:5000

Now, open another terminal in our project directory and run the client:

$ go run client.go

You should get a response on the client side like this:

Starting Client

Calling CreateUser()
User has been created: user:<uid:3 name:"Sarah Connors" nationality:"Canadian" zip:45015>  

Updating a user
A user has been updated: user:<uid:1 name:"Mandy Williams" nationality:"American" zip:10111>  

Deleting user
User with ID: 2 has been deleted.

Fetching a user
User: user:<uid:1 name:"Mandy Williams" nationality:"American" zip:10111>

From the output, we can see that our client ran successfully and all RPC calls worked as expected.

While on the server side:

Starting server...
Hosting server on: 127.0.0.1:5000
Create User
Updating User
Deleting User
User with id 2 has been deleted.
Fetching User

We can see that all print calls inside our RPC methods implementation were made on the server-side, which shows all methods were executed on the server through the calls, and responses received on the client-side.

We have successfully built a CRUD web API using gRPC and protocol buffers.

Securing Communications

In gRPC, the client and server talk over HTTP/2 with binary data (i.e. Protocol buffers); but gRPC also offers SSL/TLS integration which can be used to authenticate the server and encrypt the message exchange.

Conclusion

In this article, we have learned how to build web APIs utilizing gRPC and protocol buffers in Go.

Introduction This article describes gRPC, protocol buffers, and how to use these tools to build a simple web API in Go. Prerequisites Working knowledge of Go. Properly installed Go toolchain (Go 1.x installed). Knowledge of how APIs work. gRPC gRPC (gRPC Remote Procedure Calls) is a modern, open-source, high-performance RPC framework that can run in any environment and uses HTTP 2.0 as the underlying transport protocol. It enables transparent communication between clients and servers and simplifies the building of connected systems. During server-client communication in gRPC, as in many other RPC systems, a client application can directly invoke a method on another system - as if it was a local call, this makes it easier to build highly efficient distributed services. gRPC clients and servers can run in a variety of environments and be implemented in various supported languages; you can build a gRPC server in Go and implement clients in Ruby, Python, or any one of the other supported languages. In gRPC, you define a service to specify what methods can be called remotely with the corresponding parameters and return types. On the server-side, this interface is implemented, and a gRPC server is run to handle client calls, while on the client-side, a client that provides access to the same methods as a server - is run. As we mentioned earlier, gRPC utilizes HTTP 2.0 as the underlying transport protocol. This allows gRPC to get the most out of HTTP 2 advantages, including streaming communication and bidirectional support. Typically, in the request-response communication model built-in HTTP 1.1 used by REST APIs, if a service receives multiple requests from multiple clients - the service has to respond to each request one at a time; this, in turn, slows down the entire system, but in HTTP 2.0's client-response communication model that supports streaming and includes bidirectional support, gRPC can receive multiple requests from multiple clients, and handle those requests simultaneously by constantly streaming information. gRPC supports the following types of streams: Unary streams - a single request-response pair between client and server. Server streams - the server responds with a stream of messages to a client's request. Client streams - the client sends a stream of messages to the server, and the server, in turn, responds with a single response. Bidirectional streams - both streams (client and server) are independent of each other and can both transmit messages in any order. The client is the one who initiates and ends a bidirectional stream. By default, gRPC utilizes a binary format called Protocol Buffers to serialize data; gRPC offers very high performance and is lightweight. Protocol Buffers Protocol Buffers binary format (Protobuf) is an open-source, cross-platform data format used to serialize structured data. You define the structure of the data you want to serialize in a proto file, a normal text file with a .proto extension. Then, use the protocol buffer compiler, protoc - to generate data access classes in your preferred language using your proto definition. Protocol buffer data is structured as messages, with name-value pairs: message Student { string name = 1; int32 id = 2; bool active = 3; } These messages are used as parameters and return types for gRPC services defined in proto files, with RPC methods: // The Echo service definition service Echo { // Sends a greeting back rpc SayHi (HiRequest) returns (HiResponse) {} } // The request message containing the user’s name message HiRequest { string name = 1; } // The response message containing a greeting message HiReply { string message = 1; } In gRPC, you create a service that has RPC methods; these methods take a request message and return a response message. gRPC uses protoc with a special plugin to generate code from your proto file; it generates the gRPC server and client code, as well as the regular protocol buffer code for serialiing and retrieving your message types. Now that we've covered what gRPC and protocol buffers entail, let's get started building our gRPC web API by defining our API resources and installing the required tools. Identifying our API resources The first step to building APIs is to define what endpoints will be available via your API. We will be building a simple gRPC web API that creates or reads a user resource from our server; we have the following endpoints: API Method FetchUser: Fetch a user API Method CreateUser: Creates a user API Method UpdateUser: Updates a user API Method DeleteUser: Deletes a user Let's get started with development - by installing the required libraries (gRPC and protocol buffers). Installing gRPC To install gRPC, run the following command in your terminal: $ go get -u google.golang.org/grpc This will install the gRPC packages to your Go toolchain. Installing Protocol Buffers Next, let's install the protobuf compiler: $ sudo apt install -y protobuf-compiler Running the above command will install the protocol buffer compiler, which we will be using to generate code from our proto file, later on, using the protoc command. Creating our project Let's create our project in our GOPATH: $ mkdir $GOPATH/src/grpc-api Now cd into our project directory: $ cd $GOPATH/src/grpc-api Defining our proto file Now create a new folder for our proto files: $ mkdir protobuf Let’s create our proto file: $ touch protobuf/protobuf.proto Open our newly created file in your text editor, and add the following lines: syntax = "proto3"; package protobuf; The first line tells the protocol buffer compiler that we're using proto3 syntax, or else the compiler will assume that we are using proto2. This must be the first line non-empty, non-comment line in a proto file. Then, we specify the package this file belongs to. Next, we define our message and its field types: message User { int32 uid = 1; string name = 2; string nationality = 3; int32 zip = 4; } This specifies a User message type that takes four fields consisting of two integers (uid and zip), and two strings (name and nationality). You can also use other messages as field types. The number assigned to each field in the message definition is uniquely used to identify the field in the message binary format and should not be changed once the message type is in use. Next, let’s define our message request/response pair for our FetchUser method: message FetchUserRequest { int32 uid = 1; } message FetchUserResponse { User user = 1; } In our FetchUserRequest message, we accept a user ID (uid) that will be used to find the requested user. We'll return the requested user in a FetchUserResponse message that takes a User message type that we defined earlier as a field. Then, we define similar request/response message pairs for the rest of our methods: message CreateUserRequest { User user = 1; } message CreateUserResponse { User user = 1; } message UpdateUserRequest { User user = 1; } message UpdateUserResponse { User user = 1; } message DeleteUserRequest { int32 uid = 1; } message DeleteUserResponse { int32 uid = 1; } Above, we define CreateUserRequest and CreateUserResponse, which both take a User message type as fields. So, when a client sends a User in the CreateUserRequest, we send back the created user in a CreateUserResponse message. We also create request/response pairs for the rest of our methods, UpdateUser and DeleteUser. Now that we’ve defined our message request/response pairs, let’s create our service and register our RPC methods (FetchUser, CreateUser, UpdateUser, and DeleteUser) under the service: service UserService { rpc FetchUser(FetchUserRequest) returns (FetchUserResponse); rpc CreateUser(CreateUserRequest) returns (CreateUserResponse); rpc UpdateUser(UpdateUserRequest) returns (UpdateUserResponse); rpc DeleteUser(DeleteUserRequest) returns (DeleteUserResponse); } Here, we defined a service called UserService, and registered our RPC methods with their parameters and return values, e.g. the FetchUser method accepts a FetchUserRequest as a parameter and returns a FetchUserResponse, etc. Now our protobuf.proto file should look like this: syntax = "proto3"; package protobuf; message User { int32 uid = 1; string name = 2; string nationality = 3; int32 zip = 4; } message FetchUserRequest { int32 uid = 1; } message FetchUserResponse { User user = 1; } message CreateUserRequest { User user = 1; } message CreateUserResponse { User user = 1; } message UpdateUserRequest { User user = 1; } message UpdateUserResponse { User user = 1; } message DeleteUserRequest { int32 uid = 1; } message DeleteUserResponse { int32 uid = 1; } service UserService { rpc FetchUser(FetchUserRequest) returns (FetchUserResponse); rpc CreateUser(CreateUserRequest) returns (CreateUserResponse); rpc UpdateUser(UpdateUserRequest) returns (UpdateUserResponse); rpc DeleteUser(DeleteUserRequest) returns (DeleteUserResponse); } To compile our proto file with the protocol buffer compiler and generate the corresponding Go code to use, simply run the command from our project directory: $ protoc --gogo_out=plugins=grpc:. protobuf/protobuf.proto Running this command will generate a .pb.go file in our protobuf directory, which contains the equivalent Go bindings from our proto file which implements the gRPC code our application will use, along with server and client methods. Creating our Server We can build our server after generating Go code from our protocol buffer compiler (using protoc). Let's create our server.go file: $ touch server.go Open the file in your favorite text editor, and add the following lines: package main import ( "context" "fmt" "log" "net" "errors" "time" "math" "google.golang.org/grpc" "google.golang.org/grpc/keepalive" "grpc-api/protobuf" ) We've just imported the required libraries, including our protobuf package, so that we can access the methods and structs generated by protoc. The protocol buffer compiler generated a UserServiceServer interface for our UserService, which implements the RPC methods that we defined. Before we go on to define our server, let's create a struct that we can use as our database: type userDetails struct { Uid int32 Name string Nationality string Zip int32 } var users = []userDetails{ { Uid: 1, Name: "Josh Winters", Nationality: "American", Zip: 10111, }, { Uid: 2, Name: "Brian Stone", Nationality: "British", Zip: 20212, }, } We've created a userDetails struct with identical fields to the User type defined in our proto file; we will be using a slice of userDetails as our database for reading, updating, creating, and deleting users. Let's create helper functions to help convert between User and userDetails type: func toUser(data userDetails) *protobuf.User { return &protobuf.User { Uid: data.Uid, Name: data.Name, Nationality: data.Nationality, Zip: data.Zip, } } func fromUser(user *protobuf.User) userDetails { return userDetails { Uid: user.GetUid(), Name: user.GetName(), Nationality: user.GetNationality(), Zip: user.GetZip(), } } Now we can convert back and forth from User to userDetails easily, with our helper functions; we use the Get method to extract the values from our User type. Next, we define a server struct that will satisfy the UserServiceServer interface generated by protoc: type server struct { protobuf.UserServiceServer } Now, let’s define our methods for the server: FetchUser() method func (s *server) FetchUser(ctx context.Context, req *protobuf.FetchUserRequest) (*protobuf.FetchUserResponse, error) { fmt.Println("Fetching User") uid := req.GetUid() for _, user := range users { if user.Uid == uid { return &protobuf.FetchUserResponse { User: toUser(user), }, nil } } return nil, errors.New("User not found") } The generated FetchUser() method accepts two arguments; a context and a FetchUserRequest and returns a FetchUserResponse and an error - so to implement it we pass it a context and FetchUserRequest as arguments. In our implementation, we print a "Fetching User" message to the screen and use the GetUid() method generated by protoc in the .pb.go file to get the Uid from the FetchUserRequest. Then, we iterate through our list of users and return the user if found else we return nil and an error. CreateUser() method func (s *server) CreateUser(ctx context.Context, req *protobuf.CreateUserRequest) (*protobuf.CreateUserResponse, error) { fmt.Println("Creating User") user := req.GetUser() data := fromUser(user) users = append(users, data) return &protobuf.CreateUserResponse { User: toUser(data), }, nil } Here, we read the user from the CreateUserRequest using the GetUser() method generated by the compiler - to get the User from the request; next, we convert the user to a userDetails type using our fromUser helper function, then append the user to our list of users and send back the created user as a response. UpdateUser() method func (s *server) UpdateUser(ctx context.Context, req *protobuf.UpdateUserRequest) (*protobuf.UpdateUserResponse, error) { fmt.Println("Updating User") user := req.GetUser() data := fromUser(user) for i, user := range users { if user.Uid == data.Uid { users[i] = data return &protobuf.UpdateUserResponse { User: toUser(data), }, nil } } return nil, errors.New("Couldn't update user") } Next, we implement our UpdateUser method; first, we get the user from the UpdateUserRequest by using the GetUser() method, and iterate through our list of users searching for the corresponding Uid, if found - we update the user, and return the updated user as a response, else we return nil and an error. DeleteUser() method func (s *server) DeleteUser(ctx context.Context, req *protobuf.DeleteUserRequest) (*protobuf.DeleteUserResponse, error) { fmt.Println("Deleting User") uid := req.GetUid() var tmpUsers []userDetails for i, user := range users { if user.Uid == uid { tmpUsers = append(users[:i], users[i+1:]...) users = tmpUsers fmt.Printf("User with id %d has been deleted.\n", uid) return &protobuf.DeleteUserResponse { Uid: uid, }, nil } } return nil, errors.New("User does not exist") } We implement the DeleteUser() method, by fetching the Uid() from the request, and iterating through our list of users, looking for the matching Uid; if found, we delete the user from the list and return the deleted Uid, else we return nil and error if the corresponding user with the Uid was not found. Creating our main function Now that we have implemented our server struct which satisfies the interface by implementing all our methods, we can proceed with creating the main code to host the server: func main() { lis, err := net.Listen("tcp", "127.0.0.1:5000") if err != nil { log.Fatalf("Failed to listen: %v", err) } opts := []grpc.ServerOption{ grpc.MaxRecvMsgSize(math.MaxInt64), grpc.KeepaliveParams( keepalive.ServerParameters{ Timeout: 5 * time.Second, }, ), } s := grpc.NewServer(opts...) protobuf.RegisterUserServiceServer(s, &server{}) fmt.Println("Starting server...") fmt.Printf("Hosting server on: %s\n", lis.Addr().String()) if err := s.Serve(lis); err != nil { log.Fatalf("Failed to serve: %v", err) } } First, we create a listener on 127.0.0.1:5000 using: lis, err := net.Listen(tcp”, 127.0.0.1:5000”) We handle any errors creating the listener by calling log.Fatalf, which logs the error and calls os.Exit(). Using []grpc.ServerOptions, we create a slice of options for our server and set the maximum receive message size and server timeout. We then create our gRPC server and pass our slice of ServerOptions as a parameter: s := grpc.NewServer(opts…) After creating our gRPC server, we have to register it for our service: protobuf.RegisterUserServiceServer(s, &server{}) The RegisterUserServiceServer was generated by protoc, and it takes two arguments - a grpc.NewServer() instance, and a server struct that implements all the RPC methods added to our service; we then pass our implemented server struct as the second argument. This binds our gRPC server to our UserService. Afterward, we host the server by calling s.Serve() with our listener as a parameter, and add proper error handling. Putting it all together, our server.go script should look like this: package main import ( "context" "fmt" "log" "net" "errors" "math" "time" "google.golang.org/grpc" "google.golang.org/grpc/keepalive" "grpc-api/protobuf" ) type userDetails struct { Uid int32 Name string Nationality string Zip int32 } var users = []userDetails{ { Uid: 1, Name: "Josh Winters", Nationality: "American", Zip: 10111, }, { Uid: 2, Name: "Brian Stone", Nationality: "British", Zip: 20212, }, } func toUser(data userDetails) *protobuf.User { return &protobuf.User { Uid: data.Uid, Name: data.Name, Nationality: data.Nationality, Zip: data.Zip, } } func fromUser(user *protobuf.User) userDetails { return userDetails { Uid: user.GetUid(), Name: user.GetName(), Nationality: user.GetNationality(), Zip: user.GetZip(), } } type server struct { protobuf.UserServiceServer } func (s *server) FetchUser(ctx context.Context, req *protobuf.FetchUserRequest) (*protobuf.FetchUserResponse, error) { fmt.Println("Fetching User") uid := req.GetUid() for _, user := range users { if user.Uid == uid { return &protobuf.FetchUserResponse { User: toUser(user), }, nil } } return nil, errors.New("User not found") } func (s *server) CreateUser(ctx context.Context, req *protobuf.CreateUserRequest) (*protobuf.CreateUserResponse, error) { fmt.Println("Creating User") user := req.GetUser() data := fromUser(user) users = append(users, data) return &protobuf.CreateUserResponse { User: toUser(data), }, nil } func (s *server) UpdateUser(ctx context.Context, req *protobuf.UpdateUserRequest) (*protobuf.UpdateUserResponse, error) { fmt.Println("Updating User") user := req.GetUser() data := fromUser(user) for i, user := range users { if user.Uid == data.Uid { users[i] = data return &protobuf.UpdateUserResponse { User: toUser(data), }, nil } } return nil, errors.New("Couldn't update user") } func (s *server) DeleteUser(ctx context.Context, req *protobuf.DeleteUserRequest) (*protobuf.DeleteUserResponse, error) { fmt.Println("Deleting User") uid := req.GetUid() var tmpUsers []userDetails for i, user := range users { if user.Uid == uid { tmpUsers = append(users[:i], users[i+1:]...) users = tmpUsers fmt.Printf("User with id %d has been deleted.\n", uid) return &protobuf.DeleteUserResponse { Uid: uid, }, nil } } return nil, errors.New("User does not exist") } func main() { lis, err := net.Listen("tcp", "127.0.0.1:5000") if err != nil { log.Fatalf("Failed to listen: %v", err) } opts := []grpc.ServerOption{ grpc.MaxRecvMsgSize(math.MaxInt64), grpc.KeepaliveParams( keepalive.ServerParameters{ Timeout: 5 * time.Second, }, ), } s := grpc.NewServer(opts...) protobuf.RegisterUserServiceServer(s, &server{}) fmt.Println("Starting server...") fmt.Printf("Hosting server on: %s\n", lis.Addr().String()) if err := s.Serve(lis); err != nil { log.Fatalf("Failed to serve: %v", err) } } Now that we've built our server, let's build our client. Creating the Client In an RPC system, client calls are executed on the server like local calls; so the client is simply going to call our methods, and the server will process them. Let's create our client.go file: $ touch client.go Now, open up our newly created file in your favorite text editor, and add the following lines: package main import ( "context" "fmt" "log" "google.golang.org/grpc" "grpc-api/protobuf" ) Now we’ve imported the required packages, let’s create our main function: func main() { fmt.Println("Starting Client\n") cc, err := grpc.Dial("127.0.0.1:5000", grpc.WithInsecure()) if err != nil { log.Fatalf("Error connecting: %v", err) } // Close connection before exiting application defer cc.Close() c := protobuf.NewUserServiceClient(cc) First, we create a connection to our server address at "127.0.0.1:5000" using grpc.Dial(). grpc.Dial() takes transport security as a second parameter. Since we're connecting on the same host machine, we don't need any layer of additional security set, so we simply use grpc.WithInsecure(). Then we perform error handling and defer closing the connection. Next, we register our connection as a client for our service, using: c := protobuf.NewUserServiceClient(cc) To test our CreateUser RPC method, we simply create a new User message, and call the corresponding CreateUser() method: Calling CreateUser() // Create a user fmt.Println("Calling CreateUser()") user := &protobuf.User { Uid: 3, Name: "Sarah Connors", Nationality: "Canadian", Zip: 45015, } createUserResponse, err := c.CreateUser(context.Background(), protobuf.CreateUserRequest{User: user}) if err != nil { log.Fatalf("An error occurred: %v", err) } fmt.Printf("User has been created: %v\n\n", createUserResponse) We create a new protobuf.User object, and call the CreateUser() method with a context.Background() and our newly created User as a CreateUserRequest. Remember that our CreateUser method takes a CreateUserRequest as an argument, so we pass our declared user as a value for the User field in the CreateUserRequest body. Then we get back a CreateUserResponse, and print it to the screen. Now, let's call our other methods to make sure they are all working as expected: Calling UpdateUser() // Update a user updateUser := &protobuf.User { Uid: 1, Name: "Mandy Williams", Nationality: "American", Zip: 10111, } fmt.Println("Updating a user") updateUserResponse, err := c.UpdateUser(context.Background(), &protobuf.UpdateUserRequest{User: updateUser}) if err != nil { log.Fatalf("Error occurred updating user: %v", err) } fmt.Printf("A user has been updated: %v\n\n", updateUserResponse) Above, we framed a User message with the changes that we want made to that Uid, and called the UpdateUser method. Now the other fields of the User with Uid 1, will be changed on our server. Calling DeleteUser() fmt.Println("Deleting user") var delUid int32 = 2 deleteUserResponse, err := c.DeleteUser(context.Background(), &protobuf.DeleteUserRequest{Uid: delUid}) if err != nil { log.Fatalf("Error occurred deleting user: %v", err) } fmt.Printf("User with ID: %d has been deleted.\n\n", deleteUserResponse.GetUid()) Here we called DeleteUser() on the User with a Uid of 2. Calling FetchUser() // Get a user fmt.Println("Fetching a user") var getUid int32 = 1 fetchUserResponse, err := c.FetchUser(context.Background(), &protobuf.FetchUserRequest{Uid: getUid}) if err != nil { log.Fatalf("User not found error: %v", getUid) } fmt.Printf("User: %v\n", fetchUserResponse) } Now, to make sure our update worked, we will call FetchUser on the user with a Uid of 1, to see if the details were changed successfully to our framed request earlier. Our client.go file should now look like this: package main import ( "context" "fmt" "log" "google.golang.org/grpc" "grpc-api/protobuf" ) func main() { fmt.Println("Starting Client\n") cc, err := grpc.Dial("localhost:5000", grpc.WithInsecure()) if err != nil { log.Fatalf("Error connecting: %v", err) } // Close connection before exiting app defer cc.Close() c := protobuf.NewUserServiceClient(cc) // Create a user fmt.Println("Calling CreateUser()") user := &protobuf.User { Uid: 3, Name: "Sarah Connors", Nationality: "Canadian", Zip: 45015, } createUserResponse, err := c.CreateUser(context.Background(), &protobuf.CreateUserRequest{User: user}) if err != nil { log.Fatalf("An error occurred: %v", err) } fmt.Printf("User has been created: %v\n\n", createUserResponse) // userID := createUserResponse.GetUser().GetId() // Update a user updateUser := &protobuf.User { Uid: 1, Name: "Mandy Williams", Nationality: "American", Zip: 10111, } fmt.Println("Updating a user") updateUserResponse, err := c.UpdateUser(context.Background(), &protobuf.UpdateUserRequest{User: updateUser}) if err != nil { log.Fatalf("Error occurred updating user: %v", err) } fmt.Printf("A user has been updated: %v\n\n", updateUserResponse) fmt.Println("Deleting user") var delUid int32 = 2 deleteUserResponse, err := c.DeleteUser(context.Background(), &protobuf.DeleteUserRequest{Uid: delUid}) if err != nil { log.Fatalf("Error occurred deleting user: %v", err) } fmt.Printf("User with ID: %d has been deleted.\n\n", deleteUserResponse.GetUid()) // Get a user fmt.Println("Fetching a user") var getUid int32 = 1 fetchUserResponse, err := c.FetchUser(context.Background(), &protobuf.FetchUserRequest{Uid: getUid}) if err != nil { log.Fatalf("User not found error: %v", getUid) } fmt.Printf("User: %v\n", fetchUserResponse) } Running the Server and Client We have successfully built our gRPC server and client, now to test out both of them. Open up a terminal in our project directory and run the server first: $ go run server.go We'll get back our printed message: Starting server… Hosting server on 127.0.0.1:5000 Now, open another terminal in our project directory and run the client: $ go run client.go You should get a response on the client side like this: Starting Client Calling CreateUser() User has been created: user: Updating a user A user has been updated: user: Deleting user User with ID: 2 has been deleted. Fetching a user User: user: From the output, we can see that our client ran successfully and all RPC calls worked as expected. While on the server side: Starting server... Hosting server on: 127.0.0.1:5000 Create User Updating User Deleting User User with id 2 has been deleted. Fetching User We can see that all print calls inside our RPC methods implementation were made on the server-side, which shows all methods were executed on the server through the calls, and responses received on the client-side. We have successfully built a CRUD web API using gRPC and protocol buffers. Securing Communications In gRPC, the client and server talk over HTTP/2 with binary data (i.e. Protocol buffers); but gRPC also offers SSL/TLS integration which can be used to authenticate the server and encrypt the message exchange. Conclusion In this article, we have learned how to build web APIs utilizing gRPC and protocol buffers in Go.

Was this answer helpful?
Back

Powered by WHMCompleteSolution