Protocol Buffers, also called Protobuf, is Google's language- and platform-neutral system for serializing structured data. Serialization turns a message—a structured record of fields—into bytes for storage or transmission; parsing reconstructs the message so applications can access its fields. The system includes a schema language, generated code, language-specific runtime libraries and a serialization format, rather than being just a binary format or a network transport.
How Does Serialization Work?
In Protocol Buffers, a schema is the shared definition of a message's structure, written in a .proto file. Its author assigns each field a type, name and distinct number within that message. Binary records use the field number as the key; the corresponding message definition supplies the field's name and declared type. In a hypothetical Person message, first_name could be string field 1 and last_name string field 2; both contain text but have different identities.
At build time, protoc processes the .proto definitions to generate language-specific message code with field accessors and serialization/parsing methods. At runtime, the writing application uses that code and its language's runtime libraries to populate a message and serialize it into bytes. It writes those bytes to a file or sends them over a network connection. After retrieving or receiving them, a reading application uses generated parsing methods and its language's runtime libraries, based on the corresponding message contract, to reconstruct the message and access its field values.
With supported implementations sharing the message contract and format, a Java writer on one platform can exchange messages with a Python reader on another. Support differs across languages and environments, and some implementations come from projects outside Google.
Can the Schema Change?
Protocol Buffers allows binary-message evolution when changes follow the relevant compatibility rules. Reading compatibility does not guarantee that application logic remains correct.
Reader outcomes and field identity
| Rule or reader situation | Required constraint or reading outcome |
|---|---|
| Field identity | Required constraint or reading outcomeKeep field numbers stable: never repurpose or reuse them, and reserve deleted fields' numbers. |
| Older reader; added fields | Required constraint or reading outcomeAfter compatible schema changes, older code reading messages from an updated writer ignores newly added fields. |
| Older reader; deleted singular field | Required constraint or reading outcomeA field removed by the newer writer appears to old code with its default value. |
| Older reader; deleted repeated field | Required constraint or reading outcomeA removed repeated field—a field that can hold multiple values—appears to old code as an empty collection. |
| Newer reader; older message | Required constraint or reading outcomeNewer code can read older messages, receiving defaults for newly defined fields absent from those messages. |
Looking for liquidity, exploring on-ramp/off-ramp services, or seeking expert guidance?
Get startedHow Does It Compare With JSON and XML?
- Representation. Protobuf normally uses a binary representation, whereas JSON and XML are textual. XML's design goals favor human legibility and give terseness little priority.
- Efficiency. Binary Protocol Buffers can provide compact storage and fast parsing, but it is not always smaller or faster than JSON or XML. A smaller encoded payload means fewer bytes to transfer, not necessarily lower end-to-end latency; efficiency depends on workload and implementation, and large multidimensional floating-point arrays are one documented size-and-speed limitation of binary Protobuf.
ProtoJSON is Protobuf's canonical, schema-governed JSON representation, distinct from both arbitrary JSON and binary Protobuf. Its encoder turns a message into JSON for exchange with systems that do not support the binary format; its parser reconstructs a message from that JSON under the schema's parsing rules. Encoding and decoding messages in ProtoJSON require more CPU than doing so in binary Protobuf; the JSON representation also takes more encoded space, except in rare cases. Its evolution rules are stricter: it encodes field names and names of predefined choices (enum values), making renaming harder. Unknown fields are rejected by default; implementations may offer an option to ignore them, but generally do not forward them. Field additions or removals therefore depend on deploying schemas that recognize the written fields or using appropriate ignore handling.
How Do Service Definitions Relate to RPC?
A .proto service declaration specifies remotely callable methods and their input and output message types. In a schematic UserService, GetUser accepts a UserRequest message and returns a UserResponse message. These declarations specify the service contract; they do not implement the method or execute the request. gRPC, a remote procedure call (RPC) framework, uses Protobuf by default but allows alternatives.
gRPC compiler plugins use the .proto service definition to generate client and server APIs; application code must implement the server methods and run a server. In an illustrative successful unary call—one request and one response—the application calls a generated client method, which packages the arguments in the declared request message and sends it to the server. Server infrastructure decodes the request, executes the implemented method and encodes the response. When the returned status is OK, the client makes the response available to the calling application.
Where Is It Useful, and What Are Its Limits?
Protocol Buffers suits structured, record-like data used in communications protocols and persistent storage. For messages larger than a few megabytes, consider the memory cost of loading a whole message and retaining extra serialized copies; this is suitability guidance, not a hard format limit.
Binary messages are not inherently self-describing; full interpretation needs the corresponding schema, usually embodied in generated parsers and field accessors. The application need not reopen the .proto source for every parse, but inspecting structured values generally requires schema-aware tooling rather than simply reading text.
