resistics.decimate module¶
Module for time data decimation including classes and for the following
Definition of DecimationParameters
Performing decimation on time data
- resistics.decimate.get_eval_freqs_min(fs: float, f_min: float) numpy.ndarray[source]¶
Calculate evaluation frequencies with mimum allowable frequency
Highest frequency is nyquist / 4
- Parameters
fs (float) – Sampling frequency
f_min (float) – Minimum allowable frequency
- Returns
Array of evaluation frequencies
- Return type
np.ndarray
- Raises
ValueError – If f_min <= 0
Examples
>>> from resistics.decimate import get_eval_freqs_min >>> fs = 256 >>> get_eval_freqs_min(fs, 30) array([64. , 45.254834, 32. ]) >>> get_eval_freqs_min(fs, 128) Traceback (most recent call last): ... ValueError: Minimum frequency 128 must be > 64.0
- resistics.decimate.get_eval_freqs_size(fs: float, n_freqs: int) numpy.ndarray[source]¶
Calculate evaluation frequencies with maximum size
Highest frequency is nyquist/4
- Parameters
fs (float) – Sampling frequency
n_freqs (int) – Number of evaluation frequencies
- Returns
Array of evaluation frequencies
- Return type
np.ndarray
Examples
>>> from resistics.decimate import get_eval_freqs_size >>> fs = 256 >>> n_freqs = 3 >>> get_eval_freqs_size(fs, n_freqs) array([64. , 45.254834, 32. ])
- resistics.decimate.get_eval_freqs(fs: float, f_min: Optional[float] = None, n_freqs: Optional[int] = None) numpy.ndarray[source]¶
Get evaluation frequencies either based on size or a minimum frequency
- Parameters
fs (float) – Sampling frequency Hz
f_min (Optional[float], optional) – Minimum cutoff for evaluation frequencies, by default None
n_freqs (Optional[int], optional) – Number of evaluation frequencies, by default None
- Returns
Evaluation frequencies array
- Return type
np.ndarray
- Raises
ValueError – ValueError if both f_min and n_freqs are None
Examples
>>> from resistics.decimate import get_eval_freqs >>> get_eval_freqs(256, f_min=30) array([64. , 45.254834, 32. ]) >>> get_eval_freqs(256, n_freqs=3) array([64. , 45.254834, 32. ])
- pydantic model resistics.decimate.DecimationParameters[source]¶
Bases:
resistics.common.ResisticsModelDecimation parameters
- Parameters
fs (float) – Sampling frequency Hz
n_levels (int) – Number of levels
per_level (int) – Evaluation frequencies per level
min_samples (int) – Number of samples to under which to quit decimating
eval_df (pd.DataFrame) – The DataFrame with the decimation information
Examples
>>> from resistics.decimate import DecimationSetup >>> dec_setup = DecimationSetup(n_levels=3, per_level=2) >>> dec_params = dec_setup.run(128) >>> type(dec_params) <class 'resistics.decimate.DecimationParameters'> >>> print(dec_params.to_dataframe()) 0 1 fs factors increments decimation level 0 32.0 22.627417 128.0 1 1 1 16.0 11.313708 64.0 2 2 2 8.0 5.656854 32.0 4 2 >>> dec_params[2] [8.0, 5.65685424949238] >>> dec_params[2,1] 5.65685424949238 >>> dec_params.get_total_factor(2) 4 >>> dec_params.get_incremental_factor(2) 2
Show JSON schema
{ "title": "DecimationParameters", "description": "Decimation parameters\n\nParameters\n----------\nfs : float\n Sampling frequency Hz\nn_levels : int\n Number of levels\nper_level : int\n Evaluation frequencies per level\nmin_samples : int\n Number of samples to under which to quit decimating\neval_df : pd.DataFrame\n The DataFrame with the decimation information\n\nExamples\n--------\n>>> from resistics.decimate import DecimationSetup\n>>> dec_setup = DecimationSetup(n_levels=3, per_level=2)\n>>> dec_params = dec_setup.run(128)\n>>> type(dec_params)\n<class 'resistics.decimate.DecimationParameters'>\n>>> print(dec_params.to_dataframe())\n 0 1 fs factors increments\ndecimation level\n0 32.0 22.627417 128.0 1 1\n1 16.0 11.313708 64.0 2 2\n2 8.0 5.656854 32.0 4 2\n>>> dec_params[2]\n[8.0, 5.65685424949238]\n>>> dec_params[2,1]\n5.65685424949238\n>>> dec_params.get_total_factor(2)\n4\n>>> dec_params.get_incremental_factor(2)\n2", "type": "object", "properties": { "fs": { "title": "Fs", "type": "number" }, "n_levels": { "title": "N Levels", "type": "integer" }, "per_level": { "title": "Per Level", "type": "integer" }, "min_samples": { "title": "Min Samples", "exclusiveMinimum": 0, "type": "integer" }, "eval_freqs": { "title": "Eval Freqs", "type": "array", "items": { "type": "number" } }, "dec_factors": { "title": "Dec Factors", "type": "array", "items": { "type": "integer" } }, "dec_increments": { "title": "Dec Increments", "type": "array", "items": { "type": "integer" } }, "dec_fs": { "title": "Dec Fs", "type": "array", "items": { "type": "number" } } }, "required": [ "fs", "n_levels", "per_level", "min_samples", "eval_freqs", "dec_factors" ] }
- field fs: float [Required]¶
- field n_levels: int [Required]¶
- field per_level: int [Required]¶
- field min_samples: pydantic.types.PositiveInt [Required]¶
- Constraints
exclusiveMinimum = 0
- field eval_freqs: List[float] [Required]¶
- field dec_factors: List[int] [Required]¶
- field dec_increments: Optional[List[int]] = None¶
- Validated by
set_dec_increments
- field dec_fs: Optional[List[float]] = None¶
- Validated by
set_dec_fs
- get_eval_freqs(level: int) List[float][source]¶
Get the evaluation frequencies for a level
- Parameters
level (int) – The decimation level
- Returns
List of evaluation frequencies
- Return type
List[float]
- get_eval_freq(level: int, idx: int) float[source]¶
Get an evaluation frequency
- Parameters
level (int) – The level
idx (int) – Evaluation frequency index
- Returns
The evaluation frequency
- Return type
float
- get_fs(level: int) float[source]¶
Get sampling frequency for level
- Parameters
level (int) – The decimation level
- Returns
Sampling frequency Hz
- Return type
float
- get_total_factor(level: int) int[source]¶
Get total decimation factor for a level
- Parameters
level (int) – The level
- Returns
The decimation factor
- Return type
int
- pydantic model resistics.decimate.DecimationSetup[source]¶
Bases:
resistics.common.ResisticsProcessProcess to calculate decimation parameters
- Parameters
n_levels (int, optional) – Number of decimation levels, by default 8
per_level (int, optional) – Number of frequencies per level, by default 5
min_samples (int, optional) – Number of samples under which to quit decimating
div_factor (int, optional) – Minimum division factor for decimation, by default 2.
eval_freqs (Optional[List[float]], optional) – Explicit definition of evaluation frequencies as a flat list, by default None. Must be of size n_levels * per_level
Examples
>>> from resistics.decimate import DecimationSetup >>> dec_setup = DecimationSetup(n_levels=3, per_level=2) >>> dec_params = dec_setup.run(128) >>> print(dec_params.to_dataframe()) 0 1 fs factors increments decimation level 0 32.0 22.627417 128.0 1 1 1 16.0 11.313708 64.0 2 2 2 8.0 5.656854 32.0 4 2
Show JSON schema
{ "title": "DecimationSetup", "description": "Process to calculate decimation parameters\n\nParameters\n----------\nn_levels : int, optional\n Number of decimation levels, by default 8\nper_level : int, optional\n Number of frequencies per level, by default 5\nmin_samples : int, optional\n Number of samples under which to quit decimating\ndiv_factor : int, optional\n Minimum division factor for decimation, by default 2.\neval_freqs : Optional[List[float]], optional\n Explicit definition of evaluation frequencies as a flat list, by\n default None. Must be of size n_levels * per_level\n\nExamples\n--------\n>>> from resistics.decimate import DecimationSetup\n>>> dec_setup = DecimationSetup(n_levels=3, per_level=2)\n>>> dec_params = dec_setup.run(128)\n>>> print(dec_params.to_dataframe())\n 0 1 fs factors increments\ndecimation level\n0 32.0 22.627417 128.0 1 1\n1 16.0 11.313708 64.0 2 2\n2 8.0 5.656854 32.0 4 2", "type": "object", "properties": { "name": { "title": "Name", "type": "string" }, "n_levels": { "title": "N Levels", "default": 8, "type": "integer" }, "per_level": { "title": "Per Level", "default": 5, "type": "integer" }, "min_samples": { "title": "Min Samples", "default": 256, "type": "integer" }, "div_factor": { "title": "Div Factor", "default": 2, "type": "integer" }, "eval_freqs": { "title": "Eval Freqs", "type": "array", "items": { "type": "number" } } } }
- field n_levels: int = 8¶
- field per_level: int = 5¶
- field min_samples: int = 256¶
- field div_factor: int = 2¶
- field eval_freqs: Optional[List[float]] = None¶
- run(fs: float) resistics.decimate.DecimationParameters[source]¶
Run DecimationSetup
- Parameters
fs (float) – Sampling frequency, Hz
- Returns
Decimation parameterisation
- Return type
- pydantic model resistics.decimate.DecimatedLevelMetadata[source]¶
Bases:
resistics.common.MetadataMetadata for a decimation level
Show JSON schema
{ "title": "DecimatedLevelMetadata", "description": "Metadata for a decimation level", "type": "object", "properties": { "fs": { "title": "Fs", "type": "number" }, "n_samples": { "title": "N Samples", "type": "integer" }, "first_time": { "title": "First Time", "pattern": "%Y-%m-%d %H:%M:%S.%f_%o_%q_%v", "examples": [ "2021-01-01 00:00:00.000061_035156_250000_000000" ] }, "last_time": { "title": "Last Time", "pattern": "%Y-%m-%d %H:%M:%S.%f_%o_%q_%v", "examples": [ "2021-01-01 00:00:00.000061_035156_250000_000000" ] } }, "required": [ "fs", "n_samples", "first_time", "last_time" ] }
- field fs: float [Required]¶
The sampling frequency of the decimation level
- field n_samples: int [Required]¶
The number of samples in the decimation level
- field first_time: resistics.sampling.HighResDateTime [Required]¶
The first time in the decimation level
- Constraints
pattern = %Y-%m-%d %H:%M:%S.%f_%o_%q_%v
examples = [‘2021-01-01 00:00:00.000061_035156_250000_000000’]
- field last_time: resistics.sampling.HighResDateTime [Required]¶
The last time in the decimation level
- Constraints
pattern = %Y-%m-%d %H:%M:%S.%f_%o_%q_%v
examples = [‘2021-01-01 00:00:00.000061_035156_250000_000000’]
- property dt¶
- pydantic model resistics.decimate.DecimatedMetadata[source]¶
Bases:
resistics.common.WriteableMetadataMetadata for DecimatedData
Show JSON schema
{ "title": "DecimatedMetadata", "description": "Metadata for DecimatedData", "type": "object", "properties": { "file_info": { "$ref": "#/definitions/ResisticsFile" }, "fs": { "title": "Fs", "type": "array", "items": { "type": "number" } }, "chans": { "title": "Chans", "type": "array", "items": { "type": "string" } }, "n_chans": { "title": "N Chans", "type": "integer" }, "n_levels": { "title": "N Levels", "type": "integer" }, "first_time": { "title": "First Time", "pattern": "%Y-%m-%d %H:%M:%S.%f_%o_%q_%v", "examples": [ "2021-01-01 00:00:00.000061_035156_250000_000000" ] }, "last_time": { "title": "Last Time", "pattern": "%Y-%m-%d %H:%M:%S.%f_%o_%q_%v", "examples": [ "2021-01-01 00:00:00.000061_035156_250000_000000" ] }, "system": { "title": "System", "default": "", "type": "string" }, "serial": { "title": "Serial", "default": "", "type": "string" }, "wgs84_latitude": { "title": "Wgs84 Latitude", "default": -999.0, "type": "number" }, "wgs84_longitude": { "title": "Wgs84 Longitude", "default": -999.0, "type": "number" }, "easting": { "title": "Easting", "default": -999.0, "type": "number" }, "northing": { "title": "Northing", "default": -999.0, "type": "number" }, "elevation": { "title": "Elevation", "default": -999.0, "type": "number" }, "chans_metadata": { "title": "Chans Metadata", "type": "object", "additionalProperties": { "$ref": "#/definitions/ChanMetadata" } }, "levels_metadata": { "title": "Levels Metadata", "type": "array", "items": { "$ref": "#/definitions/DecimatedLevelMetadata" } }, "history": { "title": "History", "default": { "records": [] }, "allOf": [ { "$ref": "#/definitions/History" } ] } }, "required": [ "fs", "chans", "n_levels", "first_time", "last_time", "chans_metadata", "levels_metadata" ], "definitions": { "ResisticsFile": { "title": "ResisticsFile", "description": "Required information for writing out a resistics file", "type": "object", "properties": { "created_on_local": { "title": "Created On Local", "type": "string", "format": "date-time" }, "created_on_utc": { "title": "Created On Utc", "type": "string", "format": "date-time" }, "version": { "title": "Version", "type": "string" } } }, "ChanMetadata": { "title": "ChanMetadata", "description": "Channel metadata", "type": "object", "properties": { "name": { "title": "Name", "type": "string" }, "data_files": { "title": "Data Files", "type": "array", "items": { "type": "string" } }, "chan_type": { "title": "Chan Type", "type": "string" }, "chan_source": { "title": "Chan Source", "type": "string" }, "sensor": { "title": "Sensor", "default": "", "type": "string" }, "serial": { "title": "Serial", "default": "", "type": "string" }, "gain1": { "title": "Gain1", "default": 1, "type": "number" }, "gain2": { "title": "Gain2", "default": 1, "type": "number" }, "scaling": { "title": "Scaling", "default": 1, "type": "number" }, "chopper": { "title": "Chopper", "default": false, "type": "boolean" }, "dipole_dist": { "title": "Dipole Dist", "default": 1, "type": "number" }, "sensor_calibration_file": { "title": "Sensor Calibration File", "default": "", "type": "string" }, "instrument_calibration_file": { "title": "Instrument Calibration File", "default": "", "type": "string" } }, "required": [ "name" ] }, "DecimatedLevelMetadata": { "title": "DecimatedLevelMetadata", "description": "Metadata for a decimation level", "type": "object", "properties": { "fs": { "title": "Fs", "type": "number" }, "n_samples": { "title": "N Samples", "type": "integer" }, "first_time": { "title": "First Time", "pattern": "%Y-%m-%d %H:%M:%S.%f_%o_%q_%v", "examples": [ "2021-01-01 00:00:00.000061_035156_250000_000000" ] }, "last_time": { "title": "Last Time", "pattern": "%Y-%m-%d %H:%M:%S.%f_%o_%q_%v", "examples": [ "2021-01-01 00:00:00.000061_035156_250000_000000" ] } }, "required": [ "fs", "n_samples", "first_time", "last_time" ] }, "Record": { "title": "Record", "description": "Class to hold a record\n\nA record holds information about a process that was run. It is intended to\ntrack processes applied to data, allowing a process history to be saved\nalong with any datasets.\n\nExamples\n--------\nA simple example of creating a process record\n\n>>> from resistics.common import Record\n>>> messages = [\"message 1\", \"message 2\"]\n>>> record = Record(\n... creator={\"name\": \"example\", \"parameter1\": 15},\n... messages=messages,\n... record_type=\"example\"\n... )\n>>> record.summary()\n{\n 'time_local': '...',\n 'time_utc': '...',\n 'creator': {'name': 'example', 'parameter1': 15},\n 'messages': ['message 1', 'message 2'],\n 'record_type': 'example'\n}", "type": "object", "properties": { "time_local": { "title": "Time Local", "type": "string", "format": "date-time" }, "time_utc": { "title": "Time Utc", "type": "string", "format": "date-time" }, "creator": { "title": "Creator", "type": "object" }, "messages": { "title": "Messages", "type": "array", "items": { "type": "string" } }, "record_type": { "title": "Record Type", "type": "string" } }, "required": [ "creator", "messages", "record_type" ] }, "History": { "title": "History", "description": "Class for storing processing history\n\nParameters\n----------\nrecords : List[Record], optional\n List of records, by default []\n\nExamples\n--------\n>>> from resistics.testing import record_example1, record_example2\n>>> from resistics.common import History\n>>> record1 = record_example1()\n>>> record2 = record_example2()\n>>> history = History(records=[record1, record2])\n>>> history.summary()\n{\n 'records': [\n {\n 'time_local': '...',\n 'time_utc': '...',\n 'creator': {\n 'name': 'example1',\n 'a': 5,\n 'b': -7.0\n },\n 'messages': ['Message 1', 'Message 2'],\n 'record_type': 'process'\n },\n {\n 'time_local': '...',\n 'time_utc': '...',\n 'creator': {\n 'name': 'example2',\n 'a': 'parzen',\n 'b': -21\n },\n 'messages': ['Message 5', 'Message 6'],\n 'record_type': 'process'\n }\n ]\n}", "type": "object", "properties": { "records": { "title": "Records", "default": [], "type": "array", "items": { "$ref": "#/definitions/Record" } } } } } }
- field fs: List[float] [Required]¶
- field chans: List[str] [Required]¶
- field n_chans: Optional[int] = None¶
- Validated by
validate_n_chans
- field n_levels: int [Required]¶
- field first_time: resistics.sampling.HighResDateTime [Required]¶
- Constraints
pattern = %Y-%m-%d %H:%M:%S.%f_%o_%q_%v
examples = [‘2021-01-01 00:00:00.000061_035156_250000_000000’]
- field last_time: resistics.sampling.HighResDateTime [Required]¶
- Constraints
pattern = %Y-%m-%d %H:%M:%S.%f_%o_%q_%v
examples = [‘2021-01-01 00:00:00.000061_035156_250000_000000’]
- field system: str = ''¶
- field serial: str = ''¶
- field wgs84_latitude: float = -999.0¶
- field wgs84_longitude: float = -999.0¶
- field easting: float = -999.0¶
- field northing: float = -999.0¶
- field elevation: float = -999.0¶
- field chans_metadata: Dict[str, resistics.time.ChanMetadata] [Required]¶
- field levels_metadata: List[resistics.decimate.DecimatedLevelMetadata] [Required]¶
- field history: resistics.common.History = History(records=[])¶
- class resistics.decimate.DecimatedData(metadata: resistics.decimate.DecimatedMetadata, data: Dict[int, numpy.ndarray])[source]¶
Bases:
resistics.common.ResisticsDataData class for storing decimated data
The data for is stored in a dictionary attribute named data. The indices are integers representing the decimation level. Each decimation level is a numpy array of shape:
n_chans x n_samples
- Parameters
metadata (DecimatedMetadata) – The metadata
data (Dict[int, TimeData]) – The decimated time data
Examples
>>> import matplotlib.pyplot as plt >>> from resistics.testing import time_data_random >>> from resistics.decimate import DecimationSetup, Decimator >>> time_data = time_data_random(fs=256, n_samples=10_000) >>> dec_params = DecimationSetup(n_levels=4, per_freq=4).run(time_data.metadata.fs) >>> dec_data = Decimator().run(dec_params, time_data) >>> for level_metadata in dec_data.metadata.levels_metadata: ... level_metadata.summary() { 'fs': 256.0, 'n_samples': 10000, 'first_time': '2020-01-01 00:00:00.000000_000000_000000_000000', 'last_time': '2020-01-01 00:00:39.058593_750000_000000_000000' } { 'fs': 64.0, 'n_samples': 2500, 'first_time': '2020-01-01 00:00:00.000000_000000_000000_000000', 'last_time': '2020-01-01 00:00:39.046875_000000_000000_000000' } { 'fs': 8.0, 'n_samples': 313, 'first_time': '2020-01-01 00:00:00.000000_000000_000000_000000', 'last_time': '2020-01-01 00:00:39.000000_000000_000000_000000' } >>> for ilevel in range(0, dec_data.metadata.n_levels): ... data = dec_data.get_level(ilevel) ... plt.plot(dec_data.get_timestamps(ilevel), data[0], label=f"Level{ilevel}") >>> plt.legend(loc=3) >>> plt.tight_layout() >>> plt.show()
(Source code, png, hires.png, pdf)
- get_level(level: int) numpy.ndarray[source]¶
Get data for a decimation level
- Parameters
level (int) – The level
- Returns
The data for the decimation level
- Return type
np.ndarary
- Raises
ValueError – If level > max_level
- get_timestamps(level: int, samples: Optional[numpy.ndarray] = None, estimate: bool = True) Union[numpy.ndarray, pandas.core.indexes.datetimes.DatetimeIndex][source]¶
Get an array of timestamps
- Parameters
level (int) – The decimation level
samples (Optional[np.ndarray], optional) – If provided, timestamps are only returned for the specified samples, by default None
estimate (bool, optional) – Flag for using estimates instead of high precision datetimes, by default True
- Returns
The return dates. This will be a numpy array of RSDateTime objects if estimate is False, else it will be a pandas DatetimeIndex
- Return type
Union[np.ndarray, pd.DatetimeIndex]
- Raises
ValueError – If the level is not within bounds
- plot(max_pts: Optional[int] = 10000) plotly.graph_objs._figure.Figure[source]¶
Plot the decimated data
- Parameters
max_pts (Optional[int], optional) – The maximum number of points in any individual plot before applying lttbc downsampling, by default 10_000. If set to None, no downsampling will be applied.
- Returns
Plotly Figure
- Return type
go.Figure
- pydantic model resistics.decimate.Decimator[source]¶
Bases:
resistics.common.ResisticsProcessDecimate the time data into multiple levels
There are two options for decimation, using time data Resample or using time data Decimate. The default is to use Resample.
Show JSON schema
{ "title": "Decimator", "description": "Decimate the time data into multiple levels\n\nThere are two options for decimation, using time data Resample or using\ntime data Decimate. The default is to use Resample.", "type": "object", "properties": { "name": { "title": "Name", "type": "string" }, "resample": { "title": "Resample", "default": true, "type": "boolean" }, "max_single_factor": { "title": "Max Single Factor", "default": 3, "minimum": 3, "type": "integer" } } }
- field resample: bool = True¶
Boolean flag for using resampling instead of decimation
- field max_single_factor: resistics.decimate.ConstrainedIntValue = 3¶
Maximum single decimation factor, only used if resample is False
- Constraints
minimum = 3
- run(dec_params: resistics.decimate.DecimationParameters, time_data: resistics.time.TimeData) resistics.decimate.DecimatedData[source]¶
Decimate the TimeData
- Parameters
dec_params (DecimationParameters) – The decimation parameters
time_data (TimeData) – TimeData to decimate
- Returns
DecimatedData instance with all the decimated data
- Return type
- pydantic model resistics.decimate.DecimatedDataWriter[source]¶
Bases:
resistics.common.ResisticsWriterWriter of resistics decimated data
Show JSON schema
{ "title": "DecimatedDataWriter", "description": "Writer of resistics decimated data", "type": "object", "properties": { "name": { "title": "Name", "type": "string" }, "overwrite": { "title": "Overwrite", "default": true, "type": "boolean" } } }
- field overwrite: bool = True¶
- field name: Optional[str] [Required]¶
- Validated by
validate_name
- run(dir_path: pathlib.Path, dec_data: resistics.decimate.DecimatedData) None[source]¶
Write out DecimatedData
- Parameters
dir_path (Path) – The directory path to write to
dec_data (DecimatedData) – Decimated data to write out
- Raises
WriteError – If unable to write to the directory
- pydantic model resistics.decimate.DecimatedDataReader[source]¶
Bases:
resistics.common.ResisticsProcessReader of resistics decimated data
Show JSON schema
{ "title": "DecimatedDataReader", "description": "Reader of resistics decimated data", "type": "object", "properties": { "name": { "title": "Name", "type": "string" } } }
- field name: Optional[str] [Required]¶
- Validated by
validate_name
- run(dir_path: pathlib.Path, metadata_only: bool = False) Union[resistics.decimate.DecimatedMetadata, resistics.decimate.DecimatedData][source]¶
Read DecimatedData
- Parameters
dir_path (Path) – The directory path to read from
metadata_only (bool, optional) – Flag for getting metadata only, by default False
- Returns
DecimatedData or DecimatedMetadata if metadata_only
- Return type
Union[DecimatedMetadata, DecimatedData]
- Raises
ReadError – If the directory does not exist