In the Web Services Jungle i mean there are so many kinds of Web Services (XML RPC, SOAP, REST, …) with a lot of data formats (JSON, Protobuf, Thrift, XML,…). Services discovery is well mastered for SOAP and WSDL, REST and swagger or Thrift, but with gRPC it’s a little bit more complicated. So have a look at grpc Server reflection in go :)

In this post i will explain you, how to auto discover services on a go gRPC Server with Reflection enabled, for that i will use a nice tool gRPC CLI to introspect server protos and send test requests.

I. What are gRPC and protocol buffers (Protobuf)

1. Protobuf

Protocol buffers are Google’s language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages.

For More information on Protobuf check the Developer Guide

2. gRPC

Like many RPC systems, gRPC is based around the idea of defining a service, specifying the methods that can be called remotely with their parameters and return types. By default, gRPC uses protocol buffers as the Interface Definition Language (IDL) for describing both the service interface and the structure of the payload messages. It is possible to use other alternatives if desired. So, you can easily create a gRPC server in Java with clients in Go, Python, or Ruby.

For More information on gRPC check the Guide

II. Prerequisites

On my blog i only explain install for MacOs, if you need some help for setup on linux feel free to ask me.

1. Check go version

gRPC works only with Go 1.5 or higher, last go version 1.8.3

$ go version
go version go1.8.3 darwin/amd64

2. Install gRPC

last grpc go version 1.5.2

$ go get -u google.golang.org/grpc

3. Install Protocol Buffers

Check if you have protobuf installed, last protobuf version 3.4.0

$ protoc --version
libprotoc 3.3.2

You can install it with brew

$ brew install protobuf

4. Install protoc generator for go

$ go get -u github.com/golang/protobuf/protoc-gen-go

5. Install gRPC CLI

This is a C++ CLI, and you need gflags to build it.

$ brew install gflags
$ git clone -b $(curl -L https://grpc.io/release) https://github.com/grpc/grpc
$ cd grpc
$ git submodule update --init
$ make grpc_cli
# you may need brew install libtool

The grpc_cli is in directory bins/opt/ , copy grpc_cli in your /usr/local/bin for example

III. Basic gRPC server example

You can find this examples on github : gRPC-Service-With-Server-Reflection-in-go

1. The service definition with protobuf

In this service definition you can find : 2 methods Add and List, 3 messages, 1 enum.

goheroe.protogoheroe.proto
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
syntax = "proto3";

package superpower;

// Interface exported by the server.
service GoHeroe {
    rpc List(Filter) returns (SuperPowers) {}
    rpc Add(SuperPower) returns (SuperPowers) {}
}

// List from http://powerlisting.wikia.com/wiki/Category:Main_Categories?display=page&sort=mostvisited
enum SuperPowerCategory {
    option allow_alias = true;
    All = 0;

    Absorption = 200;
    Absorption1Alias = 200;
    Absorption2Alias = 200;
    Absorption3Alias = 200;
    Absorption4Alias = 200;
    Affinity = 201;
    AppearanceAlteration = 202;
    ArchetypalPowers = 203;

    BodilyFunctionalPowers = 300;

    CellularManipulation = 401;
    Combinations = 402;
    ConceptualPowers = 403;
    CosmicBasedPowers = 404;
    CrossoverPowers = 405;

    DeathbasedAbilities = 500;    
    DemonicPowers = 501;

    ElementalManipulation = 600;
    Empowerments = 601;
    Econt = 602;
    Enhancements = 603;

    FundamentalForces = 700;

    Intuition = 800;

    MagicalPowers = 000;
    Manipulations = 901;
    MetaPowers = 902;
    Mimicry = 903;
    MolecularManipulation = 904;
    MythologicalMimicry = 905;

    Negation = 1000;
    NonFightingPower = 1001;

    ObjectbasedPowers = 1100;
    OmnipotentPowers =  1101;
    Omniversepowers = 1102;

    PersonalPhysicalPowers = 1200;
    PhysicsBasedPowers = 1201;
    Physiology = 1202;
    Powerwithnoknownuser = 1203;
    PrimordialPowers = 1204;

    SciencePowers = 1300;
    SensoryPowers = 1301;
    SpiritualPowers = 1302;
    SupernaturalPowers = 1303;
    SuperpowerManipulation = 1304;

    TechnologybasedPowers = 1400;
    TranscendentPowers = 1401;
}

// A SuperPower is so cool
message SuperPower {
    string name = 1;
    SuperPowerCategory cat = 2; 
    bool  coolPow = 3;
}

// A List of so cool SuperPower
message SuperPowers {
    repeated SuperPower superPow = 1;
}

// An optional Filter just for fun
message Filter {
    SuperPowerCategory category = 1;
}

2. Compile it with protoc

A file goheroe.pb.go will be generated in proto directory

MakefileMakefile
1
2
$ make
protoc -I ./proto/ ./proto/goheroe.proto --go_out=plugins=grpc:proto

3. The grpc server main

As you can see we have enable reflection with the line : reflection.Register(grpcServer)

main.gomain.go
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
package main

import (
	"fmt"
	"net"

	"github.com/chemidy/goheroe-code-examples/gRPC-Service-Discovery-With-Server-Reflection-in-go/app"
	"github.com/chemidy/goheroe-code-examples/gRPC-Service-Discovery-With-Server-Reflection-in-go/proto"
	"google.golang.org/grpc"
	"google.golang.org/grpc/reflection"
)

const (
	grpcPort = ":44444"
)

func main() {

	// Creates a tcp listener on port :authGrpcPort
	lis, err := net.Listen("tcp", grpcPort)
	if err != nil {
		fmt.Printf("failed to listen: %v\n", err)
	}

	// Creates a new gRPC server
	grpcServer := grpc.NewServer()

	// get instance of GoHeroeServer
	srv, err := app.NewGoHeroeServer()
	if err != nil {
		fmt.Printf("failed to create GoHeroeServer : %s\n", err)
	}

	// register GoHeroeServer on grpc server
	superpower.RegisterGoHeroeServer(grpcServer, srv)

	// Register reflection service on gRPC server.
	reflection.Register(grpcServer)

	// start server
	fmt.Printf("starting server on port : %s\n", grpcPort)
	grpcServer.Serve(lis)
}

IV. discover gRPC services with gRPC CLI

This is time to start the server and discover services

$ go run cmd/server/main.go
starting server on port : 44444

1. grpc_cli help

grpc_cli
No command specified
  grpc_cli ls ...         ; List services
  grpc_cli call ...       ; Call method
  grpc_cli type ...       ; Print type
  grpc_cli parse ...      ; Parse message
  grpc_cli totext ...     ; Convert binary message to text
  grpc_cli tobinary ...   ; Convert text message to binary
  grpc_cli help ...       ; Print this message, or per-command usage

2. Discover services with grpc_cli ls

So just start with a simple command :

$ grpc_cli ls localhost:44444
superpower.GoHeroe
grpc.reflection.v1alpha.ServerReflection

You can have a full descriptions with the option -l

$ grpc_cli ls localhost:44444 -l
filename: goheroe.proto
package: superpower;
service GoHeroe {
  rpc List(superpower.Filter) returns (superpower.SuperPowers) {}
  rpc Add(superpower.SuperPower) returns (superpower.SuperPowers) {}
}

filename: grpc_reflection_v1alpha/reflection.proto
package: grpc.reflection.v1alpha;
service ServerReflection {
  rpc ServerReflectionInfo(stream grpc.reflection.v1alpha.ServerReflectionRequest) returns (stream grpc.reflection.v1alpha.ServerReflectionResponse) {}
}

You can filter on a service name

$ grpc_cli ls localhost:44444 superpower.GoHeroe
List
Add
$ grpc_cli ls localhost:44444 superpower.GoHeroe -l
filename: goheroe.proto
package: superpower;
service GoHeroe {
  rpc List(superpower.Filter) returns (superpower.SuperPowers) {}
  rpc Add(superpower.SuperPower) returns (superpower.SuperPowers) {}
}

You can filter on a method name

$ grpc_cli ls localhost:44444 superpower.GoHeroe.Add
Add
$ grpc_cli ls localhost:44444 superpower.GoHeroe.Add -l
  rpc Add(superpower.SuperPower) returns (superpower.SuperPowers) {}

3. Discover request and response with grpc_cli type

$ grpc_cli type localhost:44444 superpower.SuperPowers
message SuperPowers {
  repeated .superpower.SuperPower superPow = 1[json_name = "superPow"];
}
$ grpc_cli type localhost:44444 superpower.SuperPower
message SuperPower {
  string name = 1[json_name = "name"];
  .superpower.SuperPowerCategory cat = 2[json_name = "cat"];
  bool coolPow = 3[json_name = "coolPow"];
}
$ grpc_cli type localhost:44444 superpower.Filter
message Filter {
  .superpower.SuperPowerCategory category = 1[json_name = "category"];
}

The enum type is not working !!!

$ grpc_cli type localhost:44444 superpower.SuperPowerCategory
Type superpower.SuperPowerCategory not found.

4. Test the services with grpc_cli call

$ grpc_cli call localhost:44444 List ""
connecting to localhost:44444
superPow {
  name: "Flash power"
  cat: SciencePowers
}
superPow {
  name: "Superman power"
  cat: CosmicBasedPowers
}
superPow {
  name: "Spiderman power"
  cat: SupernaturalPowers
}

Rpc succeeded with OK status

By the way enum is not working

$ grpc_cli call localhost:44444 List "category:superpower.SuperPowerCategory_SciencePowers"
[libprotobuf ERROR google/protobuf/text_format.cc:287] Error parsing text-format superpower.Filter: 1:20: Unknown enumeration value of "superpower" for field "category".
Failed to parse text format to proto.

So try with the int value of the enum superpower.SuperPowerCategory_SciencePowers

grpc_cli call localhost:44444 List "category:404"
connecting to localhost:44444
superPow {
  name: "Superman power"
  cat: CosmicBasedPowers
}

Rpc succeeded with OK status
grpc_cli call localhost:44444 List "category:0"
connecting to localhost:44444
superPow {
  name: "Flash power"
  cat: SciencePowers
}
superPow {
  name: "Superman power"
  cat: CosmicBasedPowers
}
superPow {
  name: "Spiderman power"
  cat: SupernaturalPowers
}

Rpc succeeded with OK status

V. Last words about gRPC

This is just the begining of gRPC, all google apis are in gRPC now, you can see the last api documentation for google assistant.

Protbuf and gRPC are very active on GitHub, and i think gRPC gonna be a standard for API in 2018.