python緩沖區(qū)

When people who speak different languages get together and talk, they try to use a language that everyone in the group understands.

當(dāng)說(shuō)不同語(yǔ)言的人聚在一起聊天時(shí)，他們會(huì)嘗試使用小組中每個(gè)人都能理解的語(yǔ)言。

To achieve this, everyone has to translate their thoughts, which are usually in their native language, into the language of the group. This “encoding and decoding” of language, however, leads to a loss of efficiency, speed, and precision.The same concept is present in computer systems and their components. Why should we send data in XML, JSON, or any other human-readable format if there is no need for us to understand what they are talking about directly? As long as we can still translate it into a human-readable format if explicitly needed.Protocol Buffers are a way to encode data before transportation, which efficiently shrinks data blocks and therefore increases speed when sending it. It abstracts data into a language- and platform-neutral format.

為了實(shí)現(xiàn)這一目標(biāo)，每個(gè)人都必須將他們通常以其本國(guó)語(yǔ)言表達(dá)的思想翻譯成小組的語(yǔ)言。 但是，這種語(yǔ)言的“編碼和解碼”會(huì)導(dǎo)致效率，速度和精度的損失。計(jì)算機(jī)系統(tǒng)及其組件中存在相同的概念。 如果不需要我們直接了解他們?cè)谡f(shuō)什么，為什么我們應(yīng)該以XML，JSON或任何其他人類(lèi)可讀格式發(fā)送數(shù)據(jù)？ 只要明確需要，我們?nèi)匀豢梢詫⑵滢D(zhuǎn)換為人類(lèi)可讀的格式。協(xié)議緩沖區(qū)是一種在傳輸之前對(duì)數(shù)據(jù)進(jìn)行編碼的方法，它可以有效地縮小數(shù)據(jù)塊，從而提高發(fā)送數(shù)據(jù)時(shí)的速度。 它將數(shù)據(jù)抽象為與語(yǔ)言和平臺(tái)無(wú)關(guān)的格式。

目錄 (Table of Contents)

• Why do we need Protocol Buffers?

  為什么我們需要協(xié)議緩沖區(qū)？

• What are Protocol Buffers and how do they work?

  什么是協(xié)議緩沖區(qū)，它們?nèi)绾喂ぷ?#xff1f;

• Protocol Buffers in Python

  Python中的協(xié)議緩沖區(qū)

• Final notes

  最后的筆記

為什么要使用協(xié)議緩沖區(qū)？ (Why Protocol Buffers?)

The initial purpose of Protocol Buffers was to simplify the work with request/response protocols. Before ProtoBuf, Google used a different format which required additional handling of marshaling for the messages sent.

協(xié)議緩沖區(qū)的最初目的是簡(jiǎn)化請(qǐng)求/響應(yīng)協(xié)議的工作。 在ProtoBuf之前，Google使用了另一種格式，該格式需要對(duì)發(fā)送的郵件進(jìn)行其他封送處理。

In addition to that, new versions of the previous format required the developers to make sure that new versions are understood before replacing old ones, making it a hassle to work with.

除此之外，以前格式的新版本要求開(kāi)發(fā)人員在替換舊版本之前確保已理解新版本，這使使用起來(lái)很麻煩。

This overhead motivated Google to design an interface that solves precisely those problems.

這項(xiàng)開(kāi)銷(xiāo)促使Google設(shè)計(jì)了一個(gè)可以準(zhǔn)確解決這些問(wèn)題的界面。

ProtoBuf allows changes to the protocol to be introduced without breaking compatibility. Also, servers can pass around the data and execute read operations on the data without modifying its content.

ProtoBuf允許在不破壞兼容性的情況下對(duì)協(xié)議進(jìn)行更改。 此外，服務(wù)器可以傳遞數(shù)據(jù)并在不修改其內(nèi)容的情況下對(duì)數(shù)據(jù)執(zhí)行讀取操作。

Since the format is somewhat self-describing, ProtoBuf is used as a base for automatic code generation for Serializers and Deserializers.

由于格式有些自描述，因此ProtoBuf用作自動(dòng)生成序列化器和反序列化器代碼的基礎(chǔ)。

Another interesting use case is how Google uses it for short-lived Remote Procedure Calls (RPC) and to persistently store data in Bigtable. Due to their specific use case, they integrated RPC interfaces into ProtoBuf. This allows for quick and straightforward code stub generation that can be used as starting points for the actual implementation. (More on ProtoBuf RPC.)

另一個(gè)有趣的用例是Google如何將其用于短暫的遠(yuǎn)程過(guò)程調(diào)用 (RPC)并將數(shù)據(jù)持久存儲(chǔ)在Bigtable中。 由于其特定的用例，他們將RPC接口集成到ProtoBuf中。 這允許快速直接的代碼存根生成，可用作實(shí)際實(shí)現(xiàn)的起點(diǎn)。 (有關(guān)ProtoBuf RPC的更多信息。)

Other examples of where ProtoBuf can be useful are for IoT devices that are connected through mobile networks in which the amount of sent data has to be kept small or for applications in countries where high bandwidths are still rare. Sending payloads in optimized, binary formats can lead to noticeable differences in operation cost and speed.

ProtoBuf有用的其他示例是通過(guò)移動(dòng)網(wǎng)絡(luò)連接的IoT設(shè)備，其中必須將發(fā)送的數(shù)據(jù)量保持在很小的水平，或者用于那些仍很少使用高帶寬的國(guó)家/地區(qū)。 以?xún)?yōu)化的二進(jìn)制格式發(fā)送有效載荷會(huì)導(dǎo)致操作成本和速度上的明顯差異。

Using gzip compression in your HTTPS communication can further improve those metrics.

在HTTPS通信中使用gzip壓縮可以進(jìn)一步改善這些指標(biāo)。

什么是協(xié)議緩沖區(qū)，它們?nèi)绾喂ぷ?#xff1f; (What are Protocol buffers and how do they work?)

Generally speaking, Protocol Buffers are a defined interface for the serialization of structured data. It defines a normalized way to communicate, utterly independent of languages and platforms.

一般來(lái)說(shuō)，協(xié)議緩沖區(qū)是用于結(jié)構(gòu)化數(shù)據(jù)序列化的已定義接口。 它定義了一種完全獨(dú)立于語(yǔ)言和平臺(tái)的標(biāo)準(zhǔn)化通信方式。

Google advertises its ProtoBuf like this:

Google 像這樣廣告其ProtoBuf：

Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler. You define how you want your data to be structured once …

協(xié)議緩沖區(qū)是Google的與語(yǔ)言無(wú)關(guān)，與平臺(tái)無(wú)關(guān)，可擴(kuò)展的機(jī)制，用于對(duì)結(jié)構(gòu)化數(shù)據(jù)進(jìn)行序列化(例如XML)，但更小，更快，更簡(jiǎn)單。 您定義如何一次構(gòu)造數(shù)據(jù)……

The ProtoBuf interface describes the structure of the data to be sent. Payload structures are defined as “messages” in what is called Proto-Files. Those files always end with a .proto extension.For example, the basic structure of a todolist.proto file looks like this. We will also look at a complete example in the next section.

ProtoBuf接口描述了要發(fā)送的數(shù)據(jù)的結(jié)構(gòu)。 有效載荷結(jié)構(gòu)在所謂的“原始文件”中定義為“消息”。 這些文件始終以.proto 擴(kuò)展名。例如， todolist.proto文件的基本結(jié)構(gòu)如下所示。 我們還將在下一部分中查看一個(gè)完整的示例。

syntax = "proto3";// Not necessary for Python, should still be declared to avoid name collisions // in the Protocol Buffers namespace and non-Python languages package protoblog;message TodoList {// Elements of the todo list will be defined here... }

Those files are then used to generate integration classes or stubs for the language of your choice using code generators within the protoc compiler. The current version, Proto3, already supports all the major programming languages. The community supports many more in third-party open-source implementations.

然后，使用這些文件使用協(xié)議編譯器中的代碼生成器為您選擇的語(yǔ)言生成集成類(lèi)或存根。 當(dāng)前版本Proto3已經(jīng)支持所有主要的編程語(yǔ)言。 社區(qū)支持更多第三方開(kāi)放源代碼實(shí)施。

Generated classes are the core elements of Protocol Buffers. They allow the creation of elements by instantiating new messages, based on the .proto files, which are then used for serialization. We’ll look at how this is done with Python in detail in the next section.

生成的類(lèi)是協(xié)議緩沖區(qū)的核心元素。 它們?cè)试S通過(guò)實(shí)例化基于.proto文件的新消息來(lái)創(chuàng)建元素，然后將這些消息用于序列化。 在下一節(jié)中，我們將詳細(xì)介紹如何使用Python完成此操作。

Independent of the language for serialization, the messages are serialized into a non-self-describing, binary format that is pretty useless without the initial structure definition.

與用于序列化的語(yǔ)言無(wú)關(guān)，消息被序列化為非自我描述的二進(jìn)制格式，如果沒(méi)有初始結(jié)構(gòu)定義，該格式幾乎沒(méi)有用。

The binary data can then be stored, sent over the network, and used any other way human-readable data like JSON or XML is. After transmission or storage, the byte-stream can be deserialized and restored using any language-specific, compiled protobuf class we generate from the .proto file.Using Python as an example, the process could look something like this:

然后可以存儲(chǔ)二進(jìn)制數(shù)據(jù)，通過(guò)網(wǎng)絡(luò)發(fā)送和使用其他任何人類(lèi)可讀數(shù)據(jù)(如JSON或XML)的方式。 傳輸或存儲(chǔ)后，可以使用從.proto文件生成的任何特定于語(yǔ)言的已編譯protobuf類(lèi)對(duì)字節(jié)流進(jìn)行反序列化和還原。以Python為例，該過(guò)程看起來(lái)像這樣：

First, we create a new todo list and fill it with some tasks. This todo list is then serialized and sent over the network, saved in a file, or persistently stored in a database.

首先，我們創(chuàng)建一個(gè)新的待辦事項(xiàng)列表，并執(zhí)行一些任務(wù)。 然后，此待辦事項(xiàng)列表將被序列化并通過(guò)網(wǎng)絡(luò)發(fā)送，保存在文件中或永久存儲(chǔ)在數(shù)據(jù)庫(kù)中。

The sent byte stream is deserialized using the parse method of our language-specific, compiled class.Most current architectures and infrastructures, especially microservices, are based on REST, WebSockets, or GraphQL communication. However, when speed and efficiency are essential, low-level RPCs can make a huge difference.

使用特定于語(yǔ)言的已編譯類(lèi)的parse方法對(duì)發(fā)送的字節(jié)流進(jìn)行反序列化。當(dāng)前大多數(shù)體系結(jié)構(gòu)和基礎(chǔ)結(jié)構(gòu)(尤其是微服務(wù))都基于REST，WebSockets或GraphQL通信。 但是，當(dāng)速度和效率至關(guān)重要時(shí)，低級(jí)RPC可能會(huì)產(chǎn)生很大的不同。

Instead of high overhead protocols, we can use a fast and compact way to move data between the different entities into our service without wasting many resources.

代替高開(kāi)銷(xiāo)協(xié)議，我們可以使用快速而緊湊的方式在不同實(shí)體之間將數(shù)據(jù)移動(dòng)到我們的服務(wù)中，而不會(huì)浪費(fèi)很多資源。

但是，為什么還沒(méi)有在所有地方使用它呢？ (But why isn’t it used everywhere yet?)

Protocol Buffers are a bit more complicated than other, human-readable formats. This makes them comparably harder to debug and integrate into your applications.

協(xié)議緩沖區(qū)比其他人類(lèi)可讀格式要復(fù)雜一些。 這使得它們很難進(jìn)行調(diào)試和集成到您的應(yīng)用程序中。

Iteration times in engineering also tend to increase since updates in the data require updating the proto files before usage.

工程中的迭代時(shí)間也往往會(huì)增加，因?yàn)閿?shù)據(jù)更新需要在使用前更新原型文件。

Careful considerations have to be made since ProtoBuf might be an over-engineered solution in many cases.

由于ProtoBuf在許多情況下可能是過(guò)度設(shè)計(jì)的解決方案，因此必須謹(jǐn)慎考慮。

我有什么選擇？ (What alternatives do I have?)

Several projects take a similar approach to Google’s Protocol Buffers.

一些項(xiàng)目對(duì)Google的協(xié)議緩沖區(qū)采用了類(lèi)似的方法。

Google’s Flatbuffers and a third party implementation, called Cap’n Proto, are more focused on removing the parsing and unpacking step, which is necessary to access the actual data when using ProtoBufs. They have been designed explicitly for performance-critical applications, making them even faster and more memory efficient than ProtoBuf.When focusing on the RPC capabilities of ProtoBuf (used with gRPC), there are projects from other large companies like Facebook (Apache Thrift) or Microsoft (Bond protocols) that can offer alternatives.

Google的Flatbuffers和稱(chēng)為Cap'n Proto的第三方實(shí)現(xiàn)更著重于消除解析和拆包步驟，這是使用ProtoBufs時(shí)訪問(wèn)實(shí)際數(shù)據(jù)所必需的。 它們專(zhuān)為對(duì)性能至關(guān)重要的應(yīng)用程序而設(shè)計(jì)，使其比ProtoBuf更快，內(nèi)存效率更高。當(dāng)專(zhuān)注于ProtoBuf(與gRPC結(jié)合使用)的RPC功能時(shí)，Facebook等其他大型公司(Apache Thrift)或可以提供替代方案的Microsoft(債券協(xié)議)。

Python和協(xié)議緩沖區(qū) (Python and Protocol Buffers)

Python already provides some ways of data persistence using pickling. Pickling is useful in Python-only applications. It's not well suited for more complex scenarios where data sharing with other languages or changing schemas is involved.Protocol Buffers, in contrast, are developed for exactly those scenarios.The .proto files, we’ve quickly covered before, allow the user to generate code for many supported languages.

Python已經(jīng)使用酸洗提供了一些數(shù)據(jù)持久化的方法。 酸洗在僅Python的應(yīng)用程序中很有用。 它不適用于涉及與其他語(yǔ)言共享數(shù)據(jù)或更改架構(gòu)的更復(fù)雜的場(chǎng)景。相比之下， .proto正是針對(duì)這些場(chǎng)景而開(kāi)發(fā)的.proto文件，我們之前已經(jīng)快速介紹過(guò)，允許用戶(hù)生成許多受支持語(yǔ)言的代碼。

To compile the .proto file to the language class of our choice, we use protoc, the proto compiler.If you don’t have the protoc compiler installed, there are excellent guides on how to do that:

編譯.proto 文件添加到我們選擇的語(yǔ)言類(lèi)中，我們使用protoc(即proto編譯器)。如果您未安裝protoc編譯器，則有很好的指南來(lái)指導(dǎo)您：

• MacOS / Linux

  MacOS / Linux

• Windows

  視窗

Once we’ve installed protoc on our system, we can use an extended example of our todo list structure from before and generate the Python integration class from it.

一旦在系統(tǒng)上安裝了協(xié)議，就可以使用之前的待辦事項(xiàng)列表結(jié)構(gòu)的擴(kuò)展示例，并從中生成Python集成類(lèi)。

syntax = "proto3";// Not necessary for Python but should still be declared to avoid name collisions // in the Protocol Buffers namespace and non-Python languages package protoblog;// Style guide prefers prefixing enum values instead of surrounding // with an enclosing message enum TaskState {TASK_OPEN = 0;TASK_IN_PROGRESS = 1;TASK_POST_PONED = 2;TASK_CLOSED = 3;TASK_DONE = 4; }message TodoList {int32 owner_id = 1;string owner_name = 2;message ListItems {TaskState state = 1;string task = 2;string due_date = 3;}repeated ListItems todos = 3; }

Let’s take a more detailed look at the structure of the .proto file to understand it.In the first line of the proto file, we define whether we’re using Proto2 or 3. In this case, we’re using Proto3.

讓我們更詳細(xì)地了解.proto文件的結(jié)構(gòu)以了解它。在proto文件的第一行中，我們定義是使用Proto2還是3。在這種情況下，我們使用Proto3 。

The most uncommon elements of proto files are the numbers assigned to each entity of a message. Those dedicated numbers make each attribute unique and are used to identify the assigned fields in the binary encoded output.

原始文件中最不常見(jiàn)的元素是分配給消息的每個(gè)實(shí)體的編號(hào)。 這些專(zhuān)用數(shù)字使每個(gè)屬性都唯一，并用于標(biāo)識(shí)二進(jìn)制編碼輸出中的分配字段。

One important concept to grasp is that only values 1-15 are encoded with one less byte (Hex), which is useful to understand so we can assign higher numbers to the less frequently used entities. The numbers define neither the order of encoding nor the position of the given attribute in the encoded message.

要掌握的一個(gè)重要概念是，只有值1-15會(huì)用少一個(gè)字節(jié)(Hex)進(jìn)行編碼，這對(duì)于理解很有用，因此我們可以為使用頻率較低的實(shí)體分配較高的數(shù)字。 數(shù)字既不定義編碼順序 也不定義給定屬性在編碼消息中的位置。

The package definition helps prevent name clashes. In Python, packages are defined by their directory. Therefore providing a package attribute doesn’t have any effect on the generated Python code.

程序包定義有助于防止名稱(chēng)沖突。 在Python中，軟件包由其目錄定義。 因此，提供包屬性對(duì)生成的Python代碼沒(méi)有任何影響。

Please note that this should still be declared to avoid protocol buffer related name collisions and for other languages like Java.

請(qǐng)注意，對(duì)于其他語(yǔ)言(例如Java)，仍應(yīng)聲明該名稱(chēng)以避免協(xié)議緩沖區(qū)相關(guān)的名稱(chēng)沖突。

Enumerations are simple listings of possible values for a given variable.In this case, we define an Enum for the possible states of each task on the todo list.We’ll see how to use them in a bit when we look at the usage in Python.As we can see in the example, we can also nest messages inside messages.If we, for example, want to have a list of todos associated with a given todo list, we can use the repeated keyword, which is comparable to dynamically sized arrays.

枚舉是給定變量可能值的簡(jiǎn)單列表。在這種情況下，我們?yōu)榇k事項(xiàng)列表中每個(gè)任務(wù)的可能狀態(tài)定義了一個(gè)枚舉，我們將在后面的用法中看到如何使用它們。在示例中可以看到，我們也可以將消息嵌套在消息中，例如，如果我們想要與給定的待辦事項(xiàng)列表關(guān)聯(lián)的待辦事項(xiàng)列表，則可以使用重復(fù)關(guān)鍵字，該關(guān)鍵字與動(dòng)態(tài)大小的數(shù)組。

To generate usable integration code, we use the proto compiler which compiles a given .proto file into language-specific integration classes. In our case we use the --python-out argument to generate Python-specific code.

為了生成可用的集成代碼，我們使用proto編譯器，該編譯器將給定的.proto文件編譯為特定于語(yǔ)言的集成類(lèi)。 在我們的例子中，我們使用--python-out參數(shù)生成特定于Python的代碼。

protoc -I=. --python_out=. ./todolist.proto

protoc -I=. --python_out=. ./todolist.proto

In the terminal, we invoke the protocol compiler with three parameters:

在終端中，我們使用三個(gè)參數(shù)調(diào)用協(xié)議編譯器：

-I: defines the directory where we search for any dependencies (we use . which is the current directory)

-I ：定義在其中搜索任何依賴(lài)項(xiàng)的目錄(我們使用。作為當(dāng)前目錄)

--python_out: defines the location we want to generate a Python integration class in (again we use . which is the current directory)

--python_out ：定義我們要在其中生成Python集成類(lèi)的位置(再次使用。這是當(dāng)前目錄)

The last unnamed parameter defines the .proto file that will be compiled (we use the todolist.proto file in the current directory)

最后一個(gè)未命名的參數(shù)定義將要編譯的.proto文件(我們?cè)诋?dāng)前目錄中使用todolist.proto文件)

This creates a new Python file called <name_of_proto_file>_pb2.py. In our case, it is todolist_pb2.py. When taking a closer look at this file, we won’t be able to understand much about its structure immediately.

這將創(chuàng)建一個(gè)名為<name_of_proto_file> _pb2.py的新Python文件。 在我們的例子中，它是todolist_pb2.py。 當(dāng)仔細(xì)查看此文件時(shí)，我們將無(wú)法立即了解其結(jié)構(gòu)。

This is because the generator doesn’t produce direct data access elements, but further abstracts away the complexity using metaclasses and descriptors for each attribute. They describe how a class behaves instead of each instance of that class.The more exciting part is how to use this generated code to create, build, and serialize data. A straightforward integration done with our recently generated class is seen in the following:

這是因?yàn)樯善鞑粫?huì)產(chǎn)生直接的數(shù)據(jù)訪問(wèn)元素，而是會(huì)使用元類(lèi)和每個(gè)屬性的描述符進(jìn)一步簡(jiǎn)化復(fù)雜性。 它們描述了一個(gè)類(lèi)的行為方式，而不是該類(lèi)的每個(gè)實(shí)例。更令人興奮的部分是如何使用此生成的代碼來(lái)創(chuàng)建，構(gòu)建和序列化數(shù)據(jù)。 以下是與我們最近生成的類(lèi)進(jìn)行的直接集成：

import todolist_pb2 as TodoListmy_list = TodoList.TodoList() my_list.owner_id = 1234 my_list.owner_name = "Tim"first_item = my_list.todos.add() first_item.state = TodoList.TaskState.Value("TASK_DONE") first_item.task = "Test ProtoBuf for Python" first_item.due_date = "31.10.2019"print(my_list)

It merely creates a new todo list and adds one item to it. We then print the todo list element itself and can see the non-binary, non-serialized version of the data we just defined in our script.

它僅創(chuàng)建一個(gè)新的待辦事項(xiàng)列表并向其中添加一個(gè)項(xiàng)目。 然后，我們打印待辦事項(xiàng)列表元素本身，并可以看到我們剛剛在腳本中定義的數(shù)據(jù)的非二進(jìn)制，非序列化版本。

owner_id: 1234 owner_name: "Tim" todos {state: TASK_DONEtask: "Test ProtoBuf for Python"due_date: "31.10.2019" }

Each Protocol Buffer class has methods for reading and writing messages using a Protocol Buffer-specific encoding, that encodes messages into binary format.Those two methods are SerializeToString() and ParseFromString().

每個(gè)協(xié)議緩沖區(qū)類(lèi)都有使用協(xié)議緩沖區(qū)特定的編碼來(lái)讀取和寫(xiě)入消息的方法，該方法將消息編碼為二進(jìn)制格式。這兩個(gè)方法是SerializeToString()和ParseFromString() 。

import todolist_pb2 as TodoListmy_list = TodoList.TodoList() my_list.owner_id = 1234# ...with open("./serializedFile", "wb") as fd:fd.write(my_list.SerializeToString())my_list = TodoList.TodoList() with open("./serializedFile", "rb") as fd:my_list.ParseFromString(fd.read())print(my_list)

In the code example above, we write the Serialized string of bytes into a file using the wb flags.

在上面的代碼示例中，我們使用wb標(biāo)志將字節(jié)的序列化字符串寫(xiě)入文件。

Since we have already written the file, we can read back the content and Parse it using ParseFromString. ParseFromString calls on a new instance of our Serialized class using the rb flags and parses it.

由于已經(jīng)編寫(xiě)了文件，因此可以讀回內(nèi)容并使用ParseFromString對(duì)其進(jìn)行解析。 ParseFromString使用rb標(biāo)志調(diào)用序列化類(lèi)的新實(shí)例并對(duì)其進(jìn)行解析。

If we serialize this message and print it in the console, we get the byte representation which looks like this.

如果我們將此消息序列化并在控制臺(tái)中打印，我們將獲得如下所示的字節(jié)表示形式。

b'\x08\xd2\t\x12\x03Tim\x1a(\x08\x04\x12\x18Test ProtoBuf for Python\x1a\n31.10.2019'

b'\x08\xd2\t\x12\x03Tim\x1a(\x08\x04\x12\x18Test ProtoBuf for Python\x1a\n31.10.2019'

Note the b in front of the quotes. This indicates that the following string is composed of byte octets in Python.

請(qǐng)注意引號(hào)前面的b。 這表明以下字符串由Python中的字節(jié)八位字節(jié)組成。

If we directly compare this to, e.g., XML, we can see the impact ProtoBuf serialization has on the size.

如果直接將其與XML進(jìn)行比較，我們可以看到ProtoBuf序列化對(duì)大小的影響。

<todolist><owner_id>1234</owner_id><owner_name>Tim</owner_name><todos><todo><state>TASK_DONE</state><task>Test ProtoBuf for Python</task><due_date>31.10.2019</due_date></todo></todos> </todolist>

The JSON representation, non-uglified, would look like this.

未丑化的JSON表示將如下所示。

{"todoList": {"ownerId": "1234","ownerName": "Tim","todos": [{"state": "TASK_DONE","task": "Test ProtoBuf for Python","dueDate": "31.10.2019"}] } }

Judging the different formats only by the total number of bytes used, ignoring the memory needed for the overhead of formatting it, we can of course see the difference.But in addition to the memory used for the data, we also have 12 extra bytes in ProtoBuf for formatting serialized data. Comparing that to XML, we have 171 extra bytes in XML for formatting serialized data.

僅通過(guò)使用的字節(jié)總數(shù)來(lái)判斷不同的格式，而忽略格式化所需的內(nèi)存，我們當(dāng)然可以看到區(qū)別。但是除了用于數(shù)據(jù)的內(nèi)存外，我們還有12個(gè)額外的字節(jié)ProtoBuf用于格式化序列化數(shù)據(jù)。 與XML相比，我們在XML中有171個(gè)額外的字節(jié)用于格式化序列化數(shù)據(jù)。

Without Schema, we need 136 extra bytes in JSON for formatting serialized data.

沒(méi)有Schema，我們需要JSON中的136個(gè)額外字節(jié) 格式化 序列化數(shù)據(jù)。

If we’re talking about several thousands of messages sent over the network or stored on disk, ProtoBuf can make a difference.

如果我們談?wù)摰氖峭ㄟ^(guò)網(wǎng)絡(luò)發(fā)送或存儲(chǔ)在磁盤(pán)上的數(shù)千條消息，ProtoBuf可以有所作為。

However, there is a catch. The platform Auth0.com created an extensive comparison between ProtoBuf and JSON. It shows that, when compressed, the size difference between the two can be marginal (only around 9%).

但是，有一個(gè)陷阱。 Auth0.com平臺(tái)在ProtoBuf和JSON之間進(jìn)行了廣泛的比較。 它表明，壓縮后，兩者之間的大小差異可能很小(僅9％左右)。

If you’re interested in the exact numbers, please refer to the full article, which gives a detailed analysis of several factors like size and speed.

如果您對(duì)確切的數(shù)字感興趣，請(qǐng)參閱整篇文章 ，其中詳細(xì)分析了一些因素，例如大小和速度。

An interesting side note is that each data type has a default value. If attributes are not assigned or changed, they will maintain the default values. In our case, if we don’t change the TaskState of a ListItem, it has the state of “TASK_OPEN” by default. The significant advantage of this is that non-set values are not serialized, saving additional space.

一個(gè)有趣的旁注是，每種數(shù)據(jù)類(lèi)型都有一個(gè)默認(rèn)值。 如果未分配或更改屬性，則它們將保留默認(rèn)值。 在我們的情況下，如果我們不更改ListItem的TaskState，則默認(rèn)情況下其狀態(tài)為“ TASK_OPEN”。 這樣的顯著優(yōu)點(diǎn)是未設(shè)置的值不會(huì)被序列化，從而節(jié)省了額外的空間。

If we, for example, change the state of our task from TASK_DONE to TASK_OPEN, it will not be serialized.

例如，如果我們將任務(wù)的狀態(tài)從TASK_DONE更改為T(mén)ASK_OPEN，它將不會(huì)被序列化。

owner_id: 1234 owner_name: "Tim" todos {task: "Test ProtoBuf for Python"due_date: "31.10.2019" }

b'\x08\xd2\t\x12\x03Tim\x1a&\x12\x18Test ProtoBuf for Python\x1a\n31.10.2019'

b'\x08\xd2\t\x12\x03Tim\x1a&\x12\x18Test ProtoBuf for Python\x1a\n31.10.2019'

最后說(shuō)明 (Final Notes)

As we have seen, Protocol Buffers are quite handy when it comes to speed and efficiency when working with data. Due to its powerful nature, it can take some time to get used to the ProtoBuf system, even though the syntax for defining new messages is straightforward.

如我們所見(jiàn)，在處理數(shù)據(jù)時(shí)，在速度和效率方面，協(xié)議緩沖區(qū)非常方便。 由于其強(qiáng)大的特性，即使定義新消息的語(yǔ)法很簡(jiǎn)單，也要花一些時(shí)間才能習(xí)慣ProtoBuf系統(tǒng)。

As a last note, I want to point out that there were/are discussions going on about whether Protocol Buffers are “useful” for regular applications. They were developed explicitly for problems Google had in mind.If you have any questions or feedback, feel free to reach out to me on any social media like twitter or email :)

最后一點(diǎn)，我想指出的是，關(guān)于協(xié)議緩沖區(qū)是否對(duì)常規(guī)應(yīng)用程序“有用”的討論正在進(jìn)行中。 它們是專(zhuān)門(mén)針對(duì)Google遇到的問(wèn)題而開(kāi)發(fā)的。如果您有任何疑問(wèn)或反饋，請(qǐng)隨時(shí)通過(guò)Twitter或電子郵件等任何社交媒體與我聯(lián)系：)

翻譯自: https://www.freecodecamp.org/news/googles-protocol-buffers-in-python/

python緩沖區(qū)

總結(jié)

以上是生活随笔為你收集整理的python缓冲区_如何在Python中使用Google的协议缓冲区的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。

如果覺(jué)得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。