tokio.cli.darshan_bad_ost module¶
Given one or more Darshan logs containing both POSIX and Lustre counters, attempt to determine the performance each file saw and try to correlate poorly performing files with specific Lustre OSTs.
This tool first estimates per-file I/O bandwidths by dividing the total bytes read/written to each file by the time the application spent performing I/O to that file. It then uses data from Darshan’s Lustre module to map these performance estimates to the OSTs over which each file was striped. With the list of OSTs and performance measurements corresponding to each OST, the Pearson correlation coefficient is then calculated between performance and each individual OST.
Multiple Darshan logs can be passed to increase the number of observations used for correlation. This tool does not work unless the Darshan log(s) contain data from the Lustre module.
-
tokio.cli.darshan_bad_ost.correlate_ost_performance(darshan_logs)[source]¶ Generate a DataFrame containing files, performance measurements, and OST mappings and attempt to correlate performance with individual OSTs.
-
tokio.cli.darshan_bad_ost.darshanlogs_to_ost_dataframe(darshan_logs)[source]¶ Given a set of Darshan log file paths, create a dataframe containing each file, its observed performance, and a matrix of values corresponding to what fraction of that file’s contents were probably striped on each OST.
-
tokio.cli.darshan_bad_ost.estimate_darshan_perf(ranks_data)[source]¶ Calculate performance in a sideways fashion: find the longest I/O time across any rank for this file, then divide the sum of all bytes read/written by this longest io time. This function expects to receive a dict that is keyed by MPI ranks (or a single “-1” key) and whose values are dicts corresponding to Darshan POSIX counters.