4. Data Manipulation

The following section will provide you with an understanding on how to manipulate modeled data with fuddly primitives. Data manipulation is what disruptors perform (refer to Defining Specific Disruptors). This chapter will enable you to write your own disruptors or to simply perform custom manipulation of a data coming from a file, retrieved from the network (thanks to data absorption—refer to Absorption of Raw Data that Complies to the Data Model), or even generated from scratch.

4.1. Overview

To guide you over what is possible to perform, let’s consider the following data model:

 1 from framework.node import *
 2 from framework.value_types import *
 3 from framework.node_builder import *
 4
 5  example_desc = \
 6  {'name': 'ex',
 7   'contents': [
 8       {'name': 'data0',
 9        'contents': String(values=['Plip', 'Plop']) },
10
11       {'name': 'data_group',
12        'contents': [
13
14           {'name': 'len',
15            'mutable': False,
16            'contents': LEN(vt=UINT8, after_encoding=False),
17            'node_args': 'data1',
18            'absorb_csts': AbsFullCsts(contents=False)},
19
20           {'name': 'data1',
21            'contents': String(values=['Test!', 'Hello World!']) },
22
23           {'name': 'data2',
24            'qty': (1,3),
25            'semantics': ['sem1', 'sem2'],
26            'contents': UINT16_be(min=10, max=0xa0ff),
27            'alt': [
28                 {'conf': 'alt1',
29                  'contents': SINT8(values=[1,4,8])},
30                 {'conf': 'alt2',
31                  'contents': UINT16_be(min=0xeeee, max=0xff56)} ]},
32
33           {'name': 'data3',
34            'semantics': ['sem2'],
35            'sync_qty_with': 'data2',
36            'contents': UINT8(values=[30,40,50]),
37            'alt': [
38                 {'conf': 'alt1',
39                  'contents': SINT8(values=[1,4,8])}]},
40          ]},
41
42       {'name': 'data4',
43        'contents': String(values=['Red', 'Green', 'Blue']) }
44   ]}

This is what we call a data descriptor. It cannot be used directly, it should first be transformed to fuddly internal representation based on framework.node.Node. The code below shows how to perform that:

1 nb = NodeBuilder()
2 rnode = nb.create_graph_from_desc(example_desc)
3 rnode.set_env(Env())

fuddly models data as directed acyclic graphs whose terminal nodes describe the different parts of a data format (refer to Data Modeling). In order to enable elaborated manipulations it also creates a specific object to share between all the nodes some common information related to the graph: the framework.node.Env object. You should note that we create this environment object and setup the root node with it. Actually it provides all the nodes of the graph with this environment. From now on it is possible to access the environment from any node, and fuddly is now able to deal with this graph.

Note

The method framework.node_builder.NodeBuilder.create_graph_from_desc() return a framework.node.Node which is the root of the graph.

Note

When you instantiate a modeled data from a model through framework.data_model.DataModel.get_atom() as illustrated in Using fuddly Through Advanced Python Interpreter, the environment object is created for you. Likewise, when you register a data descriptor through framework.data_model.DataModel.register() (refer to Defining the Imaginary MyDF Data Model), no need to worry about the environment.

Note

The framework.node_builder.NodeBuilder object which is used to create a graph from a data descriptor is bound to the graph and should not be used for creating another graph. It contains some information on the created graph such as a dictionary of all its nodes mb.node_dico.

4.1.1. Generate Data a.k.a. Freeze a Graph

If you want to get a data from the graph you have to freeze it first as it represents many different potential data at once (actually it acts like a template). To do so, just call the method framework.node.Node.freeze() on the root node. It will provide you with a nested set of lists containing the frozen value for each node selected within the graph to provide you with a data.

What is way more interesting in the general case is obtaining a byte string of the data. For this you just have to call framework.node.Node.to_bytes() on the root node which will first freeze the graph and then flatten the nested list automatically to provide you with the byte string.

If you want to get another data from the graph you should first unfreeze it because otherwise any further call to the previous methods will give you the same value. To do that you can call the method framework.node.Node.unfreeze(). You will then be able to get a new data by freezing it again. Actually doing so will produce the next data by cycling over the possible node values (described in the graph) in a random or a determinist way (refer to Operations on Node Properties and Attributes). If you look at getting data from the graph by walking over each of its nodes independently then you should look for instance at the generic disruptor tWALK (refer to Generic Disruptors) and also to the model walker infrastructure The Model Walker Infrastructure).

By default, unfreeze will act recursively and will affect every nodes reachable from the calling one. You can unfreeze only the node on which the method is called by switching its recursive parameter.

You may want to unfreeze the graph without changing its state, just because you performed some modifications locally and want it to be taken into account when getting a new data from the graph. (refer to Node Configurations for a usage example). For that purpose, you may use the dont_change_state parameter of framework.node.Node.unfreeze() which allows to unfreeze without cycling.

Another option you may want is to unfreeze only the constraints of your graph which based on existence conditions (refer to How to Describe a Data Format Whose Parts Change Depending on Some Fields), generator and func nodes. To do so, set the reevaluate_constraints parameter to True.

To cycle over the possible node values or shapes (for non terminal nodes) a state is kept. This state is normally reset automatically when the node is exhausted in order to cycle again. can be reset thanks to the method framework.node.Node.reset_state(). In addition to resetting the node state it also unfreezes it.

Note

When a cycle over the possible node values or shapes is terminated, a notification is raised (through the linked environment object). Depending on the Finite node attribute generic disruptors will recycle the node or change to another one. Setting the Finite property on all the graph will enable you to have an end on data generation, and to avoid the generation of duplicated data.

Finally if you want to unfreeze all the node configurations (refer to Node Configurations) at once, you should call the method framework.node.Node.unfreeze_all().

4.1.2. Create Nodes with Low-Level Primitives

Instead of using the high-level API for describing a graph you can create it by using fuddly low-level primitives. Generally, you don’t need to go through that, but for specific complex situations it could provide you with what you need. To create a graph or a single node, you always have to instantiate the class framework.node.Node which enables you to set the type of content for the main node configuration (refer to Node Configurations).

Depending on the content type the constructor will call the following methods to do the job:

Note

Methods specific to the node content (framework.node.NodeInternals) can be called directly on the node itself and it will be forwarded to the content (if the method name does not match one the framework.node.Node class).

See also

If you want to learn more about the specific operations that can be performed on each kind of content (whose base class is framework.node.NodeInternals), refer to the related class, namely:

4.1.3. Cloning a Node

A graph or any node within can be cloned in order to be used anywhere else independently from the original node. To perform such an operation you should use framework.node.Node.get_clone() like in the following example:

1 rnode_copy = rnode.get_clone('mycopy')

rnode_copy is a clone of the root node of the previous graph example, and as such it is a clone of the graph. The same operation can be achieved by creating a new node and passing as a parameter the node to copy:

1 rnode_copy = Node('mycopy', base_node=rnode, new_env=True)

When you clone a node you may want to keep its current state or ignore it (that is, cloning an unfrozen graph as if it was reset). For doing so, you have to use the parameter ignore_frozen_state of the method framework.node.Node.get_clone(). By default it is set to False which means that the state is preserved during the cloning process.

4.1.4. Display a Frozen Graph

If you want to display a frozen graph (representing one data) in ASCII-art you have to call the method framework.node.Node.show() on it. For instance the following:

rnode.show()

will display a frozen graph that looks the same as the one below:

_images/ex_show.png

4.1.5. The Node Environment

The environment which should normally be the same for all the nodes of a same graph are handled by the following methods:

4.2. Search for Nodes in a Graph

Searching a graph for specific nodes can be performed in basically two ways. Depending on the criteria based on which you want to perform the search, you should use:

  • framework.node.Node.iter_nodes_by_path(): iterator that walk through all the nodes that match the graph path—you provide as a parameter—from the node on which the method is called (or None if nothing is found). The syntax defined to represent paths is similar to the one of filesystem paths. Each path are represented by a python string, where node names are separated by /’s. For instance the path from the root node of the previous data model to the node named len is:

    'ex/data_group/len'

    Note the path provided is interpreted as a regexp.

  • framework.node.Node.get_first_node_by_path(): use the previous iterator to provide the first node that match the graph path or None if nothing is found

  • framework.node.Node.get_reachable_nodes(): It is the more flexible primitive that enables to perform a search based on syntactic and/or semantic criteria. It can take several optional parameters to define your search like a graph path regexp. Unlike the previous method it always returns a list, either filled with the nodes that has been found or with nothing. You can use other kinds of criteria to be passed through the following parameters:

    • internals_criteria: To be provided with a framework.node.NodeInternalsCriteria object. This object enable you to describe the syntactic properties you look for, such as:

    • semantics_criteria: To be provided with a framework.node.NodeSemanticCriteria object. This object enable you to describe the semantic properties you look for. They are currently limited to a list of python strings.

    • owned_conf: The name of a node configuration (refer to Node Configurations) that the targeted nodes own.

    Note

    If the search is only path-based, framework.node.Node.iter_nodes_by_path() is the preferable solution as it is more efficient.

    The following code snippet illustrates the use of such criteria for retrieving all the nodes coming from the data2 description (refer to Entangled Nodes):

     1from framework.plumbing import *
 2from framework.node import *
 3from framework.value_types import *
 4
 5fmk = FmkPlumbing()
 6fmk.start()
 7
 8fmk.run_project(name='tuto')
 9
10ex_node = fmk.dm.get_atom('ex')
11ex_node.freeze()
12
13ic = NodeInternalsCriteria(mandatory_attrs=[NodeInternals.Mutable],
14                           node_kinds=[NodeInternals_TypedValue],
15                           negative_node_subkinds=[String],
16                           negative_csts=[SyncScope.Qty])
17
18sc = NodeSemanticsCriteria(mandatory_criteria=['sem1', 'sem2'])
19
20ex_node.get_reachable_nodes(internals_criteria=ic, semantics_criteria=sc,
21                            owned_conf='alt2')

    Obviously, you don’t need all these criteria for retrieving such node. It’s only for exercise.

    Note

    For abstracting away the data model from the rest of the framework, fuddly uses the specific class framework.data.Data which acts as a data container. Thus, while interacting with the different part of the framework, Node-based data (or string-based data) should be encapsulated in a framework.data.Data object.

    For instance Data(ex_node) will create an object that encapsulate ex_node. Accessing the node again is done through the property framework.data.Data.content

4.3. The Node Dictionary Interface

The framework.node.Node implements the dictionary interface, which means the following operation are possible on a node:

1node[key]           # reading operation
2
3node[key] = value   # writing operation

As a key, you can provide:

  • A path regexp (where the node on which the method is called is considered as the root) to the nodes you want to reach. A list of the nodes will be returned for the reading operation (or None if the path match nothing), and for the writing operation all the matching nodes will get the new value. The reading operation is equivalent to calling framework.node.Node.iter_nodes_by_path() on the node and providing the parameter path_regexp with your path (except the method will return a python generator instead of a list).

    The following python code snippet illustrate the access to the node named len to retrieve its byte string representation:

    1rnode['ex/data_group/len'][0].to_bytes()
2
3# same as:
4rnode.get_first_node_by_path('ex/data_group/len').to_bytes()

  • A framework.node.NodeInternalsCriteria that match the internal attributes of interest of the nodes you want to retrieve and which are reachable from the current node. It is equivalent to calling framework.node.Node.get_reachable_nodes() on the node and providing the parameter internals_criteria with your criteria object. A list will always be returned, either empty or containing the nodes of interest.

  • A framework.node.NodeSemanticsCriteria that match the internal attributes of interest of the nodes you want to retrieve and which are reachable from the current node. It is equivalent to calling framework.node.Node.get_reachable_nodes() on the node and providing the parameter semantics_criteria with the criteria object. A list will always be returned, either empty or containing the nodes of interest.

See also

To learn how to create criteria objects refer to Search for Nodes in a Graph.

As a value, you can provide:

Warning

These methods should generally be called on a frozen graph.

4.4. Change a Node

You can change the content of a specific node by absorbing a new content (refer to Byte String Absorption).

You can also temporarily change the node value of a terminal node (until the next time framework.node.Node.unfreeze() is called on it) with the method framework.node.Node.set_frozen_value() (refer to Generate Data a.k.a. Freeze a Graph).

But if you want to make some more disruptive change and change a terminal node to a non-terminal node for instance, you have two options. Either you do it from scratch and you leverage the function described in the section Create Nodes with Low-Level Primitives. For instance:

1node_to_change.set_values(value_type=String(max_sz=10))

Or you can do it by replacing the content of one node with another one. That allows you for instance to add a data from a model to another model. To illustrate this possibility let’s consider the following code that change the node data0 of our data model example with an USB STRING descriptor (yes, that does not make sense, but you can do it if you like ;).

 1 from framework.plumbing import *
 2
 3 fmk = FmkPlumbing()
 4 fmk.run_project(name='tuto')
 5
 6 usb_str = fmk.dm.get_external_atom(dm_name='usb', data_id='STR')
 7
 8 ex_node = fmk.dm.get_atom('ex')
 9
10 ex_node['ex/data0'] = usb_str  # Perform the substitution
11
12 ex_node.show()                      # Data.show() will call .show() on the embedded node

The result is shown below:

_images/ex_subst_show.png

Note

Releasing constraints (like a CRC, an offset, a length, …) of an altered data can be useful if you want fuddly to automatically recomputes the constraint for you and still comply to the model. Refer to Generate Data a.k.a. Freeze a Graph.

You can also add subnodes to non-terminal nodes through the usage of framework.node.NodeInternals_NonTerm.add(). For instance the following code snippet will add a new node after the node data2.

1data2_node = ex_node['ex/data_group/data2'][0]
2ex_node['ex/data_group$'][0].add(Node('my_node', values=['New node added']),
3                                 after=data2_node)

Thus, if ex_node before the modification is:

[0] ex [NonTerm]
 \__(1) ex/data0 [String] size=4B
 |        \_ codec=iso8859-1
 |        \_raw: b'Plip'
 \__[1] ex/data_group [NonTerm]
 |   \__[2] ex/data_group/len [GenFunc | node_args: ex/data_group/data1]
 |   |   \__(3) ex/data_group/len/cts [UINT8] size=1B
 |   |            \_ 5 (0x5)
 |   |            \_raw: b'\x05'
 |   \__(2) ex/data_group/data1 [String] size=5B
 |   |        \_ codec=iso8859-1
 |   |        \_raw: b'Test!'
 |   \__(2) ex/data_group/data2 [UINT16_be] size=2B
 |   |        \_ 10 (0xA)
 |   |        \_raw: b'\x00\n'
 |   \__(2) ex/data_group/data3 [UINT8] size=1B
 |            \_ 30 (0x1E)
 |            \_raw: b'\x1e'
 \__(1) ex/data4 [String] size=3B
          \_ codec=iso8859-1
          \_raw: b'Red'

After the modification it will be:

[0] ex [NonTerm]
 \__(1) ex/data0 [String] size=4B
 |        \_ codec=iso8859-1
 |        \_raw: b'Plip'
 \__[1] ex/data_group [NonTerm]
 |   \__[2] ex/data_group/len [GenFunc | node_args: ex/data_group/data1]
 |   |   \__(3) ex/data_group/len/cts [UINT8] size=1B
 |   |            \_ 5 (0x5)
 |   |            \_raw: b'\x05'
 |   \__(2) ex/data_group/data1 [String] size=5B
 |   |        \_ codec=iso8859-1
 |   |        \_raw: b'Test!'
 |   \__(2) ex/data_group/data2 [UINT16_be] size=2B
 |   |        \_ 10 (0xA)
 |   |        \_raw: b'\x00\n'
 |   \__(2) ex/data_group/my_node [String] size=14B
 |   |        \_ codec=iso8859-1
 |   |        \_raw: b'New node added'
 |   \__(2) ex/data_group/data3 [UINT8] size=1B
 |            \_ 30 (0x1E)
 |            \_raw: b'\x1e'
 \__(1) ex/data4 [String] size=3B
          \_ codec=iso8859-1
          \_raw: b'Red'

4.4.1. Operations on Node Properties and Attributes

The following methods enable you to retrieve the kind of content of the node. The provided answer is for the current configuration (refer to Node Configurations) if the conf parameter is not provided:

Checking if a node is frozen (refer to Generate Data a.k.a. Freeze a Graph) can be done thanks to the method:

The following methods enable you to change specific node properties or attributes:

You can test the compliance of a node with syntactic and/or semantic criteria with the method framework.node.Node.compliant_with(). Refer to the section Search for Nodes in a Graph to learn how to specify criteria.

Any object can be added to a node as a private attribute. The private object should support the __copy__ interface. To set and retrieve a private object the following methods are provided:

Node semantics can be defined to view the data model in a specific way, which boils down to be able to search for nodes based on semantic criteria (refer to Search for Nodes in a Graph). To set semantics on nodes or to retrieve them the following methods have to be used:

4.4.2. Node Configurations

Alternative node content can be added dynamically to any node of a graph. This is called a node configuration and everything that characterize a node—its type: non-terminal, terminal, generator; its attributes; its links with other nodes; and so on—are included within. A node is then a receptacle for an arbitrary number of configurations.

Note

When a node is created it gets a default configuration named MAIN.

Configuration management is based on the following methods:

In what follows, we illustrate some node configuration change based on our data model example

 1 rnode.freeze()   # We consider there is at least 2 'data2' nodes
 2
 3 # We change the configuration of the second 'data2' node
 4 rnode['ex/data_group/data2:2'][0].set_current_conf('alt1', ignore_entanglement=True)
 5 rnode['ex/data_group/data2:2'][0].unfreeze()
 6
 7 rnode.show()
 8
 9 # We change back 'data2:2' to the default configuration
10 rnode['ex/data_group/data2:2'][0].set_current_conf('MAIN', ignore_entanglement=True)
11 # We change the configuration of the first 'data2' node
12 rnode['ex/data_group/data2'][0].set_current_conf('alt1', ignore_entanglement=True)
13 # This time we unfreeze directly the parent node
14 rnode['ex/data_group$'][0].unfreeze(dont_change_state=True)
15
16 rnode.show()

See also

Refer to Entangled Nodes about the parameter ignore_entanglement.

If you want to act on a specific configuration of a node without changing first its configuration, you can leverage the conf parameter of the methods that support it. For instance, all the methods used for setting the content of a node (refer to Create Nodes with Low-Level Primitives) are configuration aware.

Note

If you need to access to the node internals (framework.node.NodeInternals) the following attributes are provided:

4.4.3. Node Corruption Infrastructure

You can also leverage the Node-corruption Infrastructure (based on hooks within the code) for handling various corruption types easily. This infrastructure is especially used by the generic disruptor tSTRUCT (refer to Generic Disruptors). This infrastructure is based on the following primitives:

The typical way to perform a corruption with this infrastructure is illustrated in what follows. This example performs a corruption that changes from the model the allowed amount for a specific node (targeted_node) of a graph (referenced by rnode) that can be created during the data generation from the graph.

1 mini = 8
2 maxi = 10
3 rnode.env.add_node_to_corrupt(targeted_node, corrupt_type=Node.CORRUPT_NODE_QTY,
4                               corrupt_op=lambda x, y: (mini, maxi))
5
6 corrupt_rnode = Node(rnode.name, base_node=rnode, ignore_frozen_state=False,
7                      new_env=True)
8 rnode.env.remove_node_to_corrupt(targeted_node)

From now on, you have still a clean graph referenced by rnode, and a corrupted one referenced by corrupt_rnode. You can now instantiate some data from corrupt_rnode that complies to an altered data model (because we change the grammar that constrains the data generation).

The corruption operations currently defined are:

4.4.4. Byte String Absorption

This feature is described in the tutorial. Refer to Absorption of Raw Data that Complies to the Data Model to learn about it. The methods which are involved in this process are:

4.5. Miscellaneous Primitives

4.6. Entangled Nodes

Node descriptors that contain the qty attribute may trigger the creation of multiple nodes based on the same description. These nodes are created in a specific way to make them react as a group. We call the nodes of such a group: entangled nodes. If you perform a modification on any one node of the group (by calling a setter on the node for instance), all the other nodes will be affected the same way.

Some node methods are immune to the entanglement, especially all the getters, others enable you to temporarily break the entanglement through the parameter ignore_entanglement.