icedb::fs::impl Namespace Reference

Typedefs
typedef std::set< sfs::path >	ExtensionsMatching_Type
typedef std::vector< std::pair< sfs::path, std::string > >	CollectedFilesRet_Type
	File path, relative mount point. More...

Functions
sfs::path	resolveSymLinks (const sfs::path &base)
	Finds out where a symbolic link points to. More...
CollectedFilesRet_Type	collectDatasetFiles (const sfs::path &base, const ExtensionsMatching_Type &valid_extensions)
sfs::path	resolveSymlinkPathandForceExists (const std::string &location)
	Like resolveSymLinks, but throw if the resulting path does not exist. More...
CollectedFilesRet_Type	collectActualHDF5files (const sfs::path &pBaseS)
	Like collectDatasetFiles for HDF5 files, but then check that these files are, indeed, HDF5 files. More...
std::string	getUniqueVROOTname ()
	Generate a unique string, used in memort-only HDF5 file trees. More...

Variables
const ExtensionsMatching_Type	common_hdf5_extensions = { ".hdf5", ".nc", ".h5", ".cdf", ".hdf" }

Detailed Description

Internal filesystem functions. These will be mase user-inaccessible in a future release.

Deprecated:

These will be moved to a private header in a future release.

Typedef Documentation

CollectedFilesRet_Type

typedef std::vector<std::pair<sfs::path, std::string> > icedb::fs::impl::CollectedFilesRet_Type

File path, relative mount point.

Definition at line 45 of file fs_backend.hpp.

ExtensionsMatching_Type

typedef std::set<sfs::path> icedb::fs::impl::ExtensionsMatching_Type

Definition at line 39 of file fs_backend.hpp.

Function Documentation

collectActualHDF5files()

CollectedFilesRet_Type icedb::fs::impl::collectActualHDF5files

(

const sfs::path &

pBaseS

)

Like collectDatasetFiles for HDF5 files, but then check that these files are, indeed, HDF5 files.

Definition at line 82 of file fs_backend.cpp.

References collectDatasetFiles().

Referenced by icedb::Databases::Database::indexDatabase(), and icedb::Databases::Database::openDatabase().

                                                                                 {
                // Find any hdf5 files underneath the current base location (recursive)
                // and create a key/value map showing where to mount these files.
                fs::impl::CollectedFilesRet_Type mountFilesCands = fs::impl::collectDatasetFiles(pBaseS);
                fs::impl::CollectedFilesRet_Type mountFiles;
                // Ensure that the candidate files are actually HDF5 files
                for (const auto & cand : mountFilesCands) {
                    // Returns >0 if a valid HDF5 file. = 0 if not. -1 on error (nonexistent file).
                    htri_t isval = H5Fis_hdf5(cand.first.string().c_str());
                    Expects(isval >= 0 && "File should exist at this point." && cand.first.string().c_str());
                    if (isval > 0)
                        mountFiles.push_back(cand);
                    if (isval == 0)
                        std::cerr << "File " << cand.first.string() << " is somehow invalid." << std::endl;
                }
                return mountFiles;
            }

Here is the call graph for this function:

Here is the caller graph for this function:

collectDatasetFiles()

CollectedFilesRet_Type icedb::fs::impl::collectDatasetFiles	(	const sfs::path &	base,
		const ExtensionsMatching_Type &	fileExtensionsToMatch = common_hdf5_extensions
	)

Find all files under the base location that have matching extensions. Used to collect files for reading.

Todo:

Once std::filesystem adds the lexically_proximate method to all compilers, change the path decomposition to the standard method.

Definition at line 26 of file fs_backend.cpp.

References resolveSymlinkPathandForceExists(), and resolveSymLinks().

Referenced by collectActualHDF5files(), and main().

            {
                std::string sBase = base.string();
                std::replace(sBase.begin(), sBase.end(), '\\', '/');

                std::vector<std::pair<sfs::path, std::string> > res;
                auto sbase = resolveSymlinkPathandForceExists(base.string());
                if (sfs::is_regular_file(sbase)) {
                    if (valid_extensions.count(base.extension()) > 0) {
                        res.push_back(std::pair<sfs::path, std::string>(sbase, "/"));
                    }
                }
                else {
                    for (const auto& p : sfs::recursive_directory_iterator(base)) {
                        auto sp = resolveSymLinks(p.path());
                        if (!sfs::exists(sp)) continue;
                        if (sfs::is_regular_file(sp)) {
                            if (valid_extensions.count(sp.extension()) > 0) {
                                std::string sP = p.path().string();
                                std::replace(sP.begin(), sP.end(), '\\', '/');
                                Expects(sP.find(sBase) == 0);
                                std::string relPath = sP.substr(sBase.length());
                                // Remove any '/'s from the beginning of the string
                                relPath = relPath.substr(relPath.find_first_not_of('/'));

                                /*
                                std::string sP = p.path().string();
                                std::replace(sP.begin(), sP.end(), '\\', '/');
                                std::string sBase = base.string();
                                //Expects(sP.substr(0, sBase.size()) == sBase);
                                Expects(sP.size() >= sBase.size() + 1);
                                std::string relpath = sP.substr(sBase.size() + 1);
                                */

                                res.push_back(std::pair<sfs::path, std::string>(p, relPath));
                            }
                        }
                    }
                }
                return res;
            }

Here is the call graph for this function:

Here is the caller graph for this function:

getUniqueVROOTname()

std::string icedb::fs::impl::getUniqueVROOTname

(

)

Generate a unique string, used in memort-only HDF5 file trees.

Definition at line 100 of file fs_backend.cpp.

                                           {
                static std::atomic<int> i{ 0 };
                int j = i++;
                std::ostringstream out;
                out << "VIRTUAL-" << j;
                return std::string(out.str());
            }

resolveSymlinkPathandForceExists()

sfs::path icedb::fs::impl::resolveSymlinkPathandForceExists

(

const std::string &

location

)

Like resolveSymLinks, but throw if the resulting path does not exist.

Definition at line 73 of file fs_backend.cpp.

References resolveSymLinks().

Referenced by collectDatasetFiles(), and icedb::Databases::Database::indexDatabase().

                                                                                {
                sfs::path res;
                sfs::path pBase(location);
                Expects(sfs::exists(pBase));
                res = fs::impl::resolveSymLinks(pBase);
                Expects(sfs::exists(res));
                return res;
            }

Here is the call graph for this function:

Here is the caller graph for this function:

resolveSymLinks()

sfs::path icedb::fs::impl::resolveSymLinks

(

const sfs::path &

base

)

Finds out where a symbolic link points to.

Definition at line 16 of file fs_backend.cpp.

Referenced by collectDatasetFiles(), icedb::Databases::Database::openDatabase(), and resolveSymlinkPathandForceExists().

            {
                sfs::path res = base;
                while (sfs::is_symlink(res) && sfs::exists(res))
                {
                    res = sfs::read_symlink(res);
                }
                return res;
            }

Here is the caller graph for this function:

Variable Documentation

common_hdf5_extensions

const ExtensionsMatching_Type icedb::fs::impl::common_hdf5_extensions = { ".hdf5", ".nc", ".h5", ".cdf", ".hdf" }

Definition at line 13 of file fs_backend.cpp.