resistics.decimate module#
Module for time data decimation including classes and for the following
Definition of DecimationParameters
Performing decimation on time data
- resistics.decimate.get_eval_freqs_min(fs: float, f_min: float) ndarray [source]#
Calculate evaluation frequencies with mimum allowable frequency
Highest frequency is nyquist / 4
- Parameters:
- Returns:
Array of evaluation frequencies
- Return type:
np.ndarray
- Raises:
ValueError – If f_min <= 0
Examples
>>> from resistics.decimate import get_eval_freqs_min >>> fs = 256 >>> get_eval_freqs_min(fs, 30) array([64. , 45.254834, 32. ]) >>> get_eval_freqs_min(fs, 128) Traceback (most recent call last): ... ValueError: Minimum frequency 128 must be > 64.0
- resistics.decimate.get_eval_freqs_size(fs: float, n_freqs: int) ndarray [source]#
Calculate evaluation frequencies with maximum size
Highest frequency is nyquist/4
- Parameters:
- Returns:
Array of evaluation frequencies
- Return type:
np.ndarray
Examples
>>> from resistics.decimate import get_eval_freqs_size >>> fs = 256 >>> n_freqs = 3 >>> get_eval_freqs_size(fs, n_freqs) array([64. , 45.254834, 32. ])
- resistics.decimate.get_eval_freqs(fs: float, f_min: float | None = None, n_freqs: int | None = None) ndarray [source]#
Get evaluation frequencies either based on size or a minimum frequency
- Parameters:
- Returns:
Evaluation frequencies array
- Return type:
np.ndarray
- Raises:
ValueError – ValueError if both f_min and n_freqs are None
Examples
>>> from resistics.decimate import get_eval_freqs >>> get_eval_freqs(256, f_min=30) array([64. , 45.254834, 32. ]) >>> get_eval_freqs(256, n_freqs=3) array([64. , 45.254834, 32. ])
- pydantic model resistics.decimate.DecimationParameters[source]#
Bases:
ResisticsModel
Decimation parameters
- Parameters:
Examples
>>> from resistics.decimate import DecimationSetup >>> dec_setup = DecimationSetup(n_levels=3, per_level=2) >>> dec_params = dec_setup.run(128) >>> type(dec_params) <class 'resistics.decimate.DecimationParameters'> >>> print(dec_params.to_dataframe()) 0 1 fs factors increments decimation level 0 32.0 22.627417 128.0 1 1 1 16.0 11.313708 64.0 2 2 2 8.0 5.656854 32.0 4 2 >>> dec_params[2] [8.0, 5.65685424949238] >>> dec_params[2,1] 5.65685424949238 >>> dec_params.get_total_factor(2) 4 >>> dec_params.get_incremental_factor(2) 2
Show JSON schema
{ "title": "DecimationParameters", "description": "Decimation parameters\n\nParameters\n----------\nfs : float\n Sampling frequency Hz\nn_levels : int\n Number of levels\nper_level : int\n Evaluation frequencies per level\nmin_samples : int\n Number of samples to under which to quit decimating\neval_df : pd.DataFrame\n The DataFrame with the decimation information\n\nExamples\n--------\n>>> from resistics.decimate import DecimationSetup\n>>> dec_setup = DecimationSetup(n_levels=3, per_level=2)\n>>> dec_params = dec_setup.run(128)\n>>> type(dec_params)\n<class 'resistics.decimate.DecimationParameters'>\n>>> print(dec_params.to_dataframe())\n 0 1 fs factors increments\ndecimation level\n0 32.0 22.627417 128.0 1 1\n1 16.0 11.313708 64.0 2 2\n2 8.0 5.656854 32.0 4 2\n>>> dec_params[2]\n[8.0, 5.65685424949238]\n>>> dec_params[2,1]\n5.65685424949238\n>>> dec_params.get_total_factor(2)\n4\n>>> dec_params.get_incremental_factor(2)\n2", "type": "object", "properties": { "fs": { "title": "Fs", "type": "number" }, "n_levels": { "title": "N Levels", "type": "integer" }, "per_level": { "title": "Per Level", "type": "integer" }, "min_samples": { "title": "Min Samples", "exclusiveMinimum": 0, "type": "integer" }, "eval_freqs": { "title": "Eval Freqs", "type": "array", "items": { "type": "number" } }, "dec_factors": { "title": "Dec Factors", "type": "array", "items": { "type": "integer" } }, "dec_increments": { "title": "Dec Increments", "type": "array", "items": { "type": "integer" } }, "dec_fs": { "title": "Dec Fs", "type": "array", "items": { "type": "number" } } }, "required": [ "fs", "n_levels", "per_level", "min_samples", "eval_freqs", "dec_factors" ] }
- field min_samples: PositiveInt [Required]#
- Constraints:
exclusiveMinimum = 0
- pydantic model resistics.decimate.DecimationSetup[source]#
Bases:
ResisticsProcess
Process to calculate decimation parameters
- Parameters:
n_levels (int, optional) – Number of decimation levels, by default 8
per_level (int, optional) – Number of frequencies per level, by default 5
min_samples (int, optional) – Number of samples under which to quit decimating
div_factor (int, optional) – Minimum division factor for decimation, by default 2.
eval_freqs (Optional[List[float]], optional) – Explicit definition of evaluation frequencies as a flat list, by default None. Must be of size n_levels * per_level
Examples
>>> from resistics.decimate import DecimationSetup >>> dec_setup = DecimationSetup(n_levels=3, per_level=2) >>> dec_params = dec_setup.run(128) >>> print(dec_params.to_dataframe()) 0 1 fs factors increments decimation level 0 32.0 22.627417 128.0 1 1 1 16.0 11.313708 64.0 2 2 2 8.0 5.656854 32.0 4 2
Show JSON schema
{ "title": "DecimationSetup", "description": "Process to calculate decimation parameters\n\nParameters\n----------\nn_levels : int, optional\n Number of decimation levels, by default 8\nper_level : int, optional\n Number of frequencies per level, by default 5\nmin_samples : int, optional\n Number of samples under which to quit decimating\ndiv_factor : int, optional\n Minimum division factor for decimation, by default 2.\neval_freqs : Optional[List[float]], optional\n Explicit definition of evaluation frequencies as a flat list, by\n default None. Must be of size n_levels * per_level\n\nExamples\n--------\n>>> from resistics.decimate import DecimationSetup\n>>> dec_setup = DecimationSetup(n_levels=3, per_level=2)\n>>> dec_params = dec_setup.run(128)\n>>> print(dec_params.to_dataframe())\n 0 1 fs factors increments\ndecimation level\n0 32.0 22.627417 128.0 1 1\n1 16.0 11.313708 64.0 2 2\n2 8.0 5.656854 32.0 4 2", "type": "object", "properties": { "name": { "title": "Name", "type": "string" }, "n_levels": { "title": "N Levels", "default": 8, "type": "integer" }, "per_level": { "title": "Per Level", "default": 5, "type": "integer" }, "min_samples": { "title": "Min Samples", "default": 256, "type": "integer" }, "div_factor": { "title": "Div Factor", "default": 2, "type": "integer" }, "eval_freqs": { "title": "Eval Freqs", "type": "array", "items": { "type": "number" } } } }
- run(fs: float) DecimationParameters [source]#
Run DecimationSetup
- Parameters:
fs (float) – Sampling frequency, Hz
- Returns:
Decimation parameterisation
- Return type:
- pydantic model resistics.decimate.DecimatedLevelMetadata[source]#
Bases:
Metadata
Metadata for a decimation level
Show JSON schema
{ "title": "DecimatedLevelMetadata", "description": "Metadata for a decimation level", "type": "object", "properties": { "fs": { "title": "Fs", "type": "number" }, "n_samples": { "title": "N Samples", "type": "integer" }, "first_time": { "title": "First Time", "pattern": "%Y-%m-%d %H:%M:%S.%f_%o_%q_%v", "examples": [ "2021-01-01 00:00:00.000061_035156_250000_000000" ] }, "last_time": { "title": "Last Time", "pattern": "%Y-%m-%d %H:%M:%S.%f_%o_%q_%v", "examples": [ "2021-01-01 00:00:00.000061_035156_250000_000000" ] } }, "required": [ "fs", "n_samples", "first_time", "last_time" ] }
- field first_time: HighResDateTime [Required]#
The first time in the decimation level
- Constraints:
pattern = %Y-%m-%d %H:%M:%S.%f_%o_%q_%v
examples = [‘2021-01-01 00:00:00.000061_035156_250000_000000’]
- field last_time: HighResDateTime [Required]#
The last time in the decimation level
- Constraints:
pattern = %Y-%m-%d %H:%M:%S.%f_%o_%q_%v
examples = [‘2021-01-01 00:00:00.000061_035156_250000_000000’]
- property dt#
- pydantic model resistics.decimate.DecimatedMetadata[source]#
Bases:
WriteableMetadata
Metadata for DecimatedData
Show JSON schema
{ "title": "DecimatedMetadata", "description": "Metadata for DecimatedData", "type": "object", "properties": { "file_info": { "$ref": "#/definitions/ResisticsFile" }, "fs": { "title": "Fs", "type": "array", "items": { "type": "number" } }, "chans": { "title": "Chans", "type": "array", "items": { "type": "string" } }, "n_chans": { "title": "N Chans", "type": "integer" }, "n_levels": { "title": "N Levels", "type": "integer" }, "first_time": { "title": "First Time", "pattern": "%Y-%m-%d %H:%M:%S.%f_%o_%q_%v", "examples": [ "2021-01-01 00:00:00.000061_035156_250000_000000" ] }, "last_time": { "title": "Last Time", "pattern": "%Y-%m-%d %H:%M:%S.%f_%o_%q_%v", "examples": [ "2021-01-01 00:00:00.000061_035156_250000_000000" ] }, "system": { "title": "System", "default": "", "type": "string" }, "serial": { "title": "Serial", "default": "", "type": "string" }, "wgs84_latitude": { "title": "Wgs84 Latitude", "default": -999.0, "type": "number" }, "wgs84_longitude": { "title": "Wgs84 Longitude", "default": -999.0, "type": "number" }, "easting": { "title": "Easting", "default": -999.0, "type": "number" }, "northing": { "title": "Northing", "default": -999.0, "type": "number" }, "elevation": { "title": "Elevation", "default": -999.0, "type": "number" }, "chans_metadata": { "title": "Chans Metadata", "type": "object", "additionalProperties": { "$ref": "#/definitions/ChanMetadata" } }, "levels_metadata": { "title": "Levels Metadata", "type": "array", "items": { "$ref": "#/definitions/DecimatedLevelMetadata" } }, "history": { "title": "History", "default": { "records": [] }, "allOf": [ { "$ref": "#/definitions/History" } ] } }, "required": [ "fs", "chans", "n_levels", "first_time", "last_time", "chans_metadata", "levels_metadata" ], "definitions": { "ResisticsFile": { "title": "ResisticsFile", "description": "Required information for writing out a resistics file", "type": "object", "properties": { "created_on_local": { "title": "Created On Local", "type": "string", "format": "date-time" }, "created_on_utc": { "title": "Created On Utc", "type": "string", "format": "date-time" }, "version": { "title": "Version", "type": "string" } } }, "ChanMetadata": { "title": "ChanMetadata", "description": "Channel metadata", "type": "object", "properties": { "name": { "title": "Name", "type": "string" }, "data_files": { "title": "Data Files", "type": "array", "items": { "type": "string" } }, "chan_type": { "title": "Chan Type", "type": "string" }, "chan_source": { "title": "Chan Source", "type": "string" }, "sensor": { "title": "Sensor", "default": "", "type": "string" }, "serial": { "title": "Serial", "default": "", "type": "string" }, "gain1": { "title": "Gain1", "default": 1, "type": "number" }, "gain2": { "title": "Gain2", "default": 1, "type": "number" }, "scaling": { "title": "Scaling", "default": 1, "type": "number" }, "chopper": { "title": "Chopper", "default": false, "type": "boolean" }, "dipole_dist": { "title": "Dipole Dist", "default": 1, "type": "number" }, "sensor_calibration_file": { "title": "Sensor Calibration File", "default": "", "type": "string" }, "instrument_calibration_file": { "title": "Instrument Calibration File", "default": "", "type": "string" } }, "required": [ "name" ] }, "DecimatedLevelMetadata": { "title": "DecimatedLevelMetadata", "description": "Metadata for a decimation level", "type": "object", "properties": { "fs": { "title": "Fs", "type": "number" }, "n_samples": { "title": "N Samples", "type": "integer" }, "first_time": { "title": "First Time", "pattern": "%Y-%m-%d %H:%M:%S.%f_%o_%q_%v", "examples": [ "2021-01-01 00:00:00.000061_035156_250000_000000" ] }, "last_time": { "title": "Last Time", "pattern": "%Y-%m-%d %H:%M:%S.%f_%o_%q_%v", "examples": [ "2021-01-01 00:00:00.000061_035156_250000_000000" ] } }, "required": [ "fs", "n_samples", "first_time", "last_time" ] }, "Record": { "title": "Record", "description": "Class to hold a record\n\nA record holds information about a process that was run. It is intended to\ntrack processes applied to data, allowing a process history to be saved\nalong with any datasets.\n\nExamples\n--------\nA simple example of creating a process record\n\n>>> from resistics.common import Record\n>>> messages = [\"message 1\", \"message 2\"]\n>>> record = Record(\n... creator={\"name\": \"example\", \"parameter1\": 15},\n... messages=messages,\n... record_type=\"example\"\n... )\n>>> record.summary()\n{\n 'time_local': '...',\n 'time_utc': '...',\n 'creator': {'name': 'example', 'parameter1': 15},\n 'messages': ['message 1', 'message 2'],\n 'record_type': 'example'\n}", "type": "object", "properties": { "time_local": { "title": "Time Local", "type": "string", "format": "date-time" }, "time_utc": { "title": "Time Utc", "type": "string", "format": "date-time" }, "creator": { "title": "Creator", "type": "object" }, "messages": { "title": "Messages", "type": "array", "items": { "type": "string" } }, "record_type": { "title": "Record Type", "type": "string" } }, "required": [ "creator", "messages", "record_type" ] }, "History": { "title": "History", "description": "Class for storing processing history\n\nParameters\n----------\nrecords : List[Record], optional\n List of records, by default []\n\nExamples\n--------\n>>> from resistics.testing import record_example1, record_example2\n>>> from resistics.common import History\n>>> record1 = record_example1()\n>>> record2 = record_example2()\n>>> history = History(records=[record1, record2])\n>>> history.summary()\n{\n 'records': [\n {\n 'time_local': '...',\n 'time_utc': '...',\n 'creator': {\n 'name': 'example1',\n 'a': 5,\n 'b': -7.0\n },\n 'messages': ['Message 1', 'Message 2'],\n 'record_type': 'process'\n },\n {\n 'time_local': '...',\n 'time_utc': '...',\n 'creator': {\n 'name': 'example2',\n 'a': 'parzen',\n 'b': -21\n },\n 'messages': ['Message 5', 'Message 6'],\n 'record_type': 'process'\n }\n ]\n}", "type": "object", "properties": { "records": { "title": "Records", "default": [], "type": "array", "items": { "$ref": "#/definitions/Record" } } } } } }
- field first_time: HighResDateTime [Required]#
- Constraints:
pattern = %Y-%m-%d %H:%M:%S.%f_%o_%q_%v
examples = [‘2021-01-01 00:00:00.000061_035156_250000_000000’]
- field last_time: HighResDateTime [Required]#
- Constraints:
pattern = %Y-%m-%d %H:%M:%S.%f_%o_%q_%v
examples = [‘2021-01-01 00:00:00.000061_035156_250000_000000’]
- field chans_metadata: Dict[str, ChanMetadata] [Required]#
- field levels_metadata: List[DecimatedLevelMetadata] [Required]#
- class resistics.decimate.DecimatedData(metadata: DecimatedMetadata, data: Dict[int, ndarray])[source]#
Bases:
ResisticsData
Data class for storing decimated data
The data for is stored in a dictionary attribute named data. The indices are integers representing the decimation level. Each decimation level is a numpy array of shape:
n_chans x n_samples
- Parameters:
metadata (DecimatedMetadata) – The metadata
Examples
>>> import matplotlib.pyplot as plt >>> from resistics.testing import time_data_random >>> from resistics.decimate import DecimationSetup, Decimator >>> time_data = time_data_random(fs=256, n_samples=10_000) >>> dec_params = DecimationSetup(n_levels=4, per_freq=4).run(time_data.metadata.fs) >>> dec_data = Decimator().run(dec_params, time_data) >>> for level_metadata in dec_data.metadata.levels_metadata: ... level_metadata.summary() { 'fs': 256.0, 'n_samples': 10000, 'first_time': '2020-01-01 00:00:00.000000_000000_000000_000000', 'last_time': '2020-01-01 00:00:39.058593_750000_000000_000000' } { 'fs': 64.0, 'n_samples': 2500, 'first_time': '2020-01-01 00:00:00.000000_000000_000000_000000', 'last_time': '2020-01-01 00:00:39.046875_000000_000000_000000' } { 'fs': 8.0, 'n_samples': 313, 'first_time': '2020-01-01 00:00:00.000000_000000_000000_000000', 'last_time': '2020-01-01 00:00:39.000000_000000_000000_000000' } >>> for ilevel in range(0, dec_data.metadata.n_levels): ... data = dec_data.get_level(ilevel) ... plt.plot(dec_data.get_timestamps(ilevel), data[0], label=f"Level{ilevel}") >>> plt.legend(loc=3) >>> plt.tight_layout() >>> plt.show()
(
Source code
,png
,hires.png
,pdf
)- get_level(level: int) ndarray [source]#
Get data for a decimation level
- Parameters:
level (int) – The level
- Returns:
The data for the decimation level
- Return type:
np.ndarary
- Raises:
ValueError – If level > max_level
- get_timestamps(level: int, samples: ndarray | None = None, estimate: bool = True) ndarray | DatetimeIndex [source]#
Get an array of timestamps
- Parameters:
- Returns:
The return dates. This will be a numpy array of RSDateTime objects if estimate is False, else it will be a pandas DatetimeIndex
- Return type:
Union[np.ndarray, pd.DatetimeIndex]
- Raises:
ValueError – If the level is not within bounds
- plot(max_pts: int | None = 10000) Figure [source]#
Plot the decimated data
- Parameters:
max_pts (Optional[int], optional) – The maximum number of points in any individual plot before applying lttbc downsampling, by default 10_000. If set to None, no downsampling will be applied.
- Returns:
Plotly Figure
- Return type:
go.Figure
- pydantic model resistics.decimate.Decimator[source]#
Bases:
ResisticsProcess
Decimate the time data into multiple levels
There are two options for decimation, using time data Resample or using time data Decimate. The default is to use Resample.
Show JSON schema
{ "title": "Decimator", "description": "Decimate the time data into multiple levels\n\nThere are two options for decimation, using time data Resample or using\ntime data Decimate. The default is to use Resample.", "type": "object", "properties": { "name": { "title": "Name", "type": "string" }, "resample": { "title": "Resample", "default": true, "type": "boolean" }, "max_single_factor": { "title": "Max Single Factor", "default": 3, "minimum": 3, "type": "integer" } } }
- field max_single_factor: ConstrainedIntValue = 3#
Maximum single decimation factor, only used if resample is False
- Constraints:
minimum = 3
- run(dec_params: DecimationParameters, time_data: TimeData) DecimatedData [source]#
Decimate the TimeData
- Parameters:
dec_params (DecimationParameters) – The decimation parameters
time_data (TimeData) – TimeData to decimate
- Returns:
DecimatedData instance with all the decimated data
- Return type:
- pydantic model resistics.decimate.DecimatedDataWriter[source]#
Bases:
ResisticsWriter
Writer of resistics decimated data
Show JSON schema
{ "title": "DecimatedDataWriter", "description": "Writer of resistics decimated data", "type": "object", "properties": { "name": { "title": "Name", "type": "string" }, "overwrite": { "title": "Overwrite", "default": true, "type": "boolean" } } }
- run(dir_path: Path, dec_data: DecimatedData) None [source]#
Write out DecimatedData
- Parameters:
dir_path (Path) – The directory path to write to
dec_data (DecimatedData) – Decimated data to write out
- Raises:
WriteError – If unable to write to the directory
- pydantic model resistics.decimate.DecimatedDataReader[source]#
Bases:
ResisticsProcess
Reader of resistics decimated data
Show JSON schema
{ "title": "DecimatedDataReader", "description": "Reader of resistics decimated data", "type": "object", "properties": { "name": { "title": "Name", "type": "string" } } }
- run(dir_path: Path, metadata_only: bool = False) DecimatedMetadata | DecimatedData [source]#
Read DecimatedData
- Parameters:
dir_path (Path) – The directory path to read from
metadata_only (bool, optional) – Flag for getting metadata only, by default False
- Returns:
DecimatedData or DecimatedMetadata if metadata_only
- Return type:
Union[DecimatedMetadata, DecimatedData]
- Raises:
ReadError – If the directory does not exist