malib.backend.datapool.data_array module
- class malib.backend.datapool.data_array.DataArray(name: str, capacity: Optional[int] = 30000, init_capacity: int = 1000000)[source]
Bases:
abc.ABC
- class malib.backend.datapool.data_array.LinkDataArray(name: str, capacity: Optional[int] = 30000, init_capacity: Optional[int] = 1000000, block_size: Optional[int] = 25000)[source]
Bases:
malib.backend.datapool.data_array.DataArraySegmented data structure for efficient rollout data storage/inference
- property capacity: int
- class deque
Bases:
objectdeque([iterable[, maxlen]]) –> deque object
A list-like sequence optimized for data accesses near its endpoints.
- append()
Add an element to the right side of the deque.
- appendleft()
Add an element to the left side of the deque.
- clear()
Remove all elements from the deque.
- copy()
Return a shallow copy of a deque.
- count(value) integer -- return number of occurrences of value
- extend()
Extend the right side of the deque with elements from the iterable
- extendleft()
Extend the left side of the deque with elements from the iterable
- index(value[, start[, stop]]) integer -- return first index of value.
Raises ValueError if the value is not present.
- insert()
D.insert(index, object) – insert object before index
- maxlen
maximum size of a deque or None if unbounded
- pop()
Remove and return the rightmost element.
- popleft()
Remove and return the leftmost element.
- remove()
D.remove(value) – remove first occurrence of value.
- reverse()
D.reverse() – reverse IN PLACE
- rotate()
Rotate the deque n steps to the right (default n=1). If n is negative, rotates left.
- fill(data: numpy.ndarray, capacity: Optional[int] = None) malib.backend.datapool.data_array.LinkDataArray[source]
- property size: int
- class malib.backend.datapool.data_array.NumpyDataArray(name: str, capacity: Optional[int] = 30000, init_capacity: int = 1000000)[source]
Bases:
malib.backend.datapool.data_array.DataArrayData storage in the form of numpy array. The instantiation of its physical storage space is done when the first data insert/fill operation is called. Doubling its current capacity until hit the maximum capacity limitation when an insertion is called on a full data array.
- Parameters
name (str) – A parameter help users identifying the data array, will set to be consistent with the episode column names by default.
capacity (int) – The max capacity that the data array can span. The default value malib.settings.DEFAULT_EPISODE_CAPACITY is applied when *capacity* is set to be None.
init_capacity (int) – Then init size of the internal data array created when the first insert/fill operation is called.
- property capacity: int
- fill(data: numpy.ndarray, capacity: Optional[int] = None) malib.backend.datapool.data_array.NumpyDataArray[source]
Flush fill the array with the input data.
- Note: For performance issue, it is designed that after the filling, the internal data storage will only use a
shallow copy of the input data. If users are not sure if the input data will be modified afterwards, please use the deep copy of the data instead.
- Parameters
data (DataTransferType) – Array-like data input.
capacity (int) – Override the pre-set max capacity if provided.
- Returns
A reference of the caller.
- get_data() numpy.ndarray[source]
Return the existing data in the internal storage, order preserving.
:return DataTransferType
- insert(data: numpy.ndarray) None[source]
Insert data at the back of the array. A certain extent of time consumptions are expected due to: (1) The first insert/fill operation will instantiate the physical storage space. (2) Possible memory allocation & data movement when insert on a full array.
- Parameters
data (DataTransferType) – Data block to be inserted at the back of the array.
- Returns
None
- property nbytes: int
Return the memory size(in bytes) of the space occupied by the array data.
- Returns
int
- property size: int