portage.package.ebuild.fetch module¶

class portage.package.ebuild.fetch.ContentHashLayout(algo, cutoffs)¶

Bases: portage.package.ebuild.fetch.FilenameHashLayout

The content-hash layout is identical to the filename-hash layout, except for these three differences:

A content digest is used instead of a filename digest.

2) The final element of the path returned from the get_path method corresponds to the complete content digest. The path is a function of the content digest alone.

3) Because the path is a function of content digest alone, the get_filenames implementation cannot derive distfiles names from paths, so it instead yields DistfileName instances whose names are equal to content digest values. The DistfileName documentation discusses resulting implications.

Motivations to use the content-hash layout instead of the filename-hash layout may include:

1) Since the file path is independent of the file name, file name collisions cannot occur. This makes the content-hash layout suitable for storage of multiple types of files (not only gentoo distfiles). For example, it can be used to store distfiles for multiple linux distros within the same tree, with automatic deduplication based on content digest. This layout can be used to store and distribute practically anything (including binary packages for example).

2) Allows multiple revisions for the same distfiles name. An existing distfile can be updated, and if a user still has an older copy of an ebuild repository (or an overlay), then a user can successfully fetch a desired revision of the distfile as long as it has not been purged from the mirror.

3) File integrity data is integrated into the layout itself, making it very simple to verify the integrity of any file that it contains. The only tool required is an implementation of the chosen hash algorithm.

get_filenames(distdir)¶: Yields DistfileName instances each with filename corresponding to a digest value for self.algo, and which can be compared to other DistfileName instances with their digests_equal method.

get_path(filename)¶: For content-hash, the path is a function of the content digest alone. The final element of the path returned from the get_path method corresponds to the complete content digest.

static verify_args(args, filename=None)¶

If the filename argument is given, then supported hash algorithms are constrained by digests available in the filename digests attribute.

Parameters

args – layout.conf entry args
filename – filename with digests attribute

Returns

True if args are valid for available digest algorithms, and False otherwise

class portage.package.ebuild.fetch.DistfileName(s, digests=None)¶

Bases: str

The DistfileName type represents a distfile name and associated content digests, used by MirrorLayoutConfig and related layout implementations.

The path of a distfile within a layout must be dependent on nothing more than the distfile name and its associated content digests. For filename-hash layout, path is dependent on distfile name alone, and the get_filenames implementation yields strings corresponding to distfile names. For content-hash layout, path is dependent on content digest alone, and the get_filenames implementation yields DistfileName instances whose names are equal to content digest values. The content-hash layout simply lacks the filename-hash layout’s innate ability to translate a distfile path to a distfile name, and instead caries an innate ability to translate a distfile path to a content digest.

In order to prepare for a migration from filename-hash to content-hash layout, all consumers of the layout get_filenames method need to be updated to work with content digests as a substitute for distfile names. For example, emirrordist requires the –content-db option when working with a content-hash layout, which serves as a means to associate distfile names with content digest values yielded by the content-hash get_filenames implementation.

capitalize()¶

Return a capitalized version of the string.

More specifically, make the first character have upper case and the rest lower case.

casefold()¶: Return a version of the string suitable for caseless comparisons.

center(width, fillchar=' ', /)¶

Return a centered string of length width.

Padding is done using the specified fill character (default is a space).

count(sub[, start[, end]]) → int¶: Return the number of non-overlapping occurrences of substring sub in string S[start:end]. Optional arguments start and end are interpreted as in slice notation.

digests_equal(other)¶: Test if digests compare equal to those of another instance.

encode(encoding='utf-8', errors='strict')¶

Encode the string using the codec registered for encoding.

encoding: The encoding in which to encode the string.
errors: The error handling scheme to use for encoding errors. The default is ‘strict’ meaning that encoding errors raise a UnicodeEncodeError. Other possible values are ‘ignore’, ‘replace’ and ‘xmlcharrefreplace’ as well as any other name registered with codecs.register_error that can handle UnicodeEncodeErrors.

endswith(suffix[, start[, end]]) → bool¶: Return True if S ends with the specified suffix, False otherwise. With optional start, test S beginning at that position. With optional end, stop comparing S at that position. suffix can also be a tuple of strings to try.

expandtabs(tabsize=8)¶

Return a copy where all tab characters are expanded using spaces.

If tabsize is not given, a tab size of 8 characters is assumed.

find(sub[, start[, end]]) → int¶

Return the lowest index in S where substring sub is found, such that sub is contained within S[start:end]. Optional arguments start and end are interpreted as in slice notation.

Return -1 on failure.

format(*args, **kwargs) → str¶: Return a formatted version of S, using substitutions from args and kwargs. The substitutions are identified by braces (‘{’ and ‘}’).

format_map(mapping) → str¶: Return a formatted version of S, using substitutions from mapping. The substitutions are identified by braces (‘{’ and ‘}’).

index(sub[, start[, end]]) → int¶

Return the lowest index in S where substring sub is found, such that sub is contained within S[start:end]. Optional arguments start and end are interpreted as in slice notation.

Raises ValueError when the substring is not found.

isalnum()¶

Return True if the string is an alpha-numeric string, False otherwise.

A string is alpha-numeric if all characters in the string are alpha-numeric and there is at least one character in the string.

isalpha()¶

Return True if the string is an alphabetic string, False otherwise.

A string is alphabetic if all characters in the string are alphabetic and there is at least one character in the string.

isascii()¶

Return True if all characters in the string are ASCII, False otherwise.

ASCII characters have code points in the range U+0000-U+007F. Empty string is ASCII too.

isdecimal()¶

Return True if the string is a decimal string, False otherwise.

A string is a decimal string if all characters in the string are decimal and there is at least one character in the string.

isdigit()¶

Return True if the string is a digit string, False otherwise.

A string is a digit string if all characters in the string are digits and there is at least one character in the string.

isidentifier()¶

Return True if the string is a valid Python identifier, False otherwise.

Call keyword.iskeyword(s) to test whether string s is a reserved identifier, such as “def” or “class”.

islower()¶

Return True if the string is a lowercase string, False otherwise.

A string is lowercase if all cased characters in the string are lowercase and there is at least one cased character in the string.

isnumeric()¶

Return True if the string is a numeric string, False otherwise.

A string is numeric if all characters in the string are numeric and there is at least one character in the string.

isprintable()¶

Return True if the string is printable, False otherwise.

A string is printable if all of its characters are considered printable in repr() or if it is empty.

isspace()¶

Return True if the string is a whitespace string, False otherwise.

A string is whitespace if all characters in the string are whitespace and there is at least one character in the string.

istitle()¶

Return True if the string is a title-cased string, False otherwise.

In a title-cased string, upper- and title-case characters may only follow uncased characters and lowercase characters only cased ones.

isupper()¶

Return True if the string is an uppercase string, False otherwise.

A string is uppercase if all cased characters in the string are uppercase and there is at least one cased character in the string.

join(iterable, /)¶

Concatenate any number of strings.

The string whose method is called is inserted in between each given string. The result is returned as a new string.

Example: ‘.’.join([‘ab’, ‘pq’, ‘rs’]) -> ‘ab.pq.rs’

ljust(width, fillchar=' ', /)¶

Return a left-justified string of length width.

Padding is done using the specified fill character (default is a space).

lower()¶: Return a copy of the string converted to lowercase.

lstrip(chars=None, /)¶

Return a copy of the string with leading whitespace removed.

If chars is given and not None, remove characters in chars instead.

static maketrans()¶

Return a translation table usable for str.translate().

If there is only one argument, it must be a dictionary mapping Unicode ordinals (integers) or characters to Unicode ordinals, strings or None. Character keys will be then converted to ordinals. If there are two arguments, they must be strings of equal length, and in the resulting dictionary, each character in x will be mapped to the character at the same position in y. If there is a third argument, it must be a string, whose characters will be mapped to None in the result.

partition(sep, /)¶

Partition the string into three parts using the given separator.

This will search for the separator in the string. If the separator is found, returns a 3-tuple containing the part before the separator, the separator itself, and the part after it.

If the separator is not found, returns a 3-tuple containing the original string and two empty strings.

removeprefix(prefix, /)¶

Return a str with the given prefix string removed if present.

If the string starts with the prefix string, return string[len(prefix):]. Otherwise, return a copy of the original string.

removesuffix(suffix, /)¶

Return a str with the given suffix string removed if present.

If the string ends with the suffix string and that suffix is not empty, return string[:-len(suffix)]. Otherwise, return a copy of the original string.

replace(old, new, count=- 1, /)¶

Return a copy with all occurrences of substring old replaced by new.

count
Maximum number of occurrences to replace. -1 (the default value) means replace all occurrences.

If the optional argument count is given, only the first count occurrences are replaced.

rfind(sub[, start[, end]]) → int¶

Return the highest index in S where substring sub is found, such that sub is contained within S[start:end]. Optional arguments start and end are interpreted as in slice notation.

Return -1 on failure.

rindex(sub[, start[, end]]) → int¶

Return the highest index in S where substring sub is found, such that sub is contained within S[start:end]. Optional arguments start and end are interpreted as in slice notation.

Raises ValueError when the substring is not found.

rjust(width, fillchar=' ', /)¶

Return a right-justified string of length width.

Padding is done using the specified fill character (default is a space).

rpartition(sep, /)¶

Partition the string into three parts using the given separator.

This will search for the separator in the string, starting at the end. If the separator is found, returns a 3-tuple containing the part before the separator, the separator itself, and the part after it.

If the separator is not found, returns a 3-tuple containing two empty strings and the original string.

rsplit(sep=None, maxsplit=- 1)¶

Return a list of the substrings in the string, using sep as the separator string.

sep
The separator used to split the string.

When set to None (the default value), will split on any whitespace character (including \n \r \t \f and spaces) and will discard empty strings from the result.

maxsplit
Maximum number of splits (starting from the left). -1 (the default value) means no limit.

Splitting starts at the end of the string and works to the front.

rstrip(chars=None, /)¶

Return a copy of the string with trailing whitespace removed.

If chars is given and not None, remove characters in chars instead.

split(sep=None, maxsplit=- 1)¶

Return a list of the substrings in the string, using sep as the separator string.

sep
The separator used to split the string.

When set to None (the default value), will split on any whitespace character (including \n \r \t \f and spaces) and will discard empty strings from the result.

maxsplit
Maximum number of splits (starting from the left). -1 (the default value) means no limit.

Note, str.split() is mainly useful for data that has been intentionally delimited. With natural text that includes punctuation, consider using the regular expression module.

splitlines(keepends=False)¶

Return a list of the lines in the string, breaking at line boundaries.

Line breaks are not included in the resulting list unless keepends is given and true.

startswith(prefix[, start[, end]]) → bool¶: Return True if S starts with the specified prefix, False otherwise. With optional start, test S beginning at that position. With optional end, stop comparing S at that position. prefix can also be a tuple of strings to try.

strip(chars=None, /)¶

Return a copy of the string with leading and trailing whitespace removed.

If chars is given and not None, remove characters in chars instead.

swapcase()¶: Convert uppercase characters to lowercase and lowercase characters to uppercase.

title()¶

Return a version of the string where each word is titlecased.

More specifically, words start with uppercased characters and all remaining cased characters have lower case.

translate(table, /)¶

Replace each character in the string using the given translation table.

table
Translation table, which must be a mapping of Unicode ordinals to Unicode ordinals, strings, or None.

The table must implement lookup/indexing via __getitem__, for instance a dictionary or list. If this operation raises LookupError, the character is left untouched. Characters mapped to None are deleted.

upper()¶: Return a copy of the string converted to uppercase.

zfill(width, /)¶

Pad a numeric string with zeros on the left, to fill a field of the given width.

The string is never truncated.

class portage.package.ebuild.fetch.FilenameHashLayout(algo, cutoffs)¶

Bases: object

get_filenames(distdir)¶

get_path(filename)¶

static verify_args(args)¶

class portage.package.ebuild.fetch.FlatLayout¶

Bases: object

get_filenames(distdir)¶

get_path(filename)¶

static verify_args(args)¶

class portage.package.ebuild.fetch.MirrorLayoutConfig¶

Bases: object

Class to read layout.conf from a mirror.

deserialize(data)¶

get_all_layouts()¶

get_best_supported_layout(filename=None)¶

If the filename argument is given, then acceptable hash algorithms are constrained by digests available in the filename digests attribute.

Parameters: filename – filename with digests attribute

read_from_file(f)¶

serialize()¶

static validate_structure(val, filename=None)¶

If the filename argument is given, then supported hash algorithms are constrained by digests available in the filename digests attribute.

Parameters

val – layout.conf entry args
filename – filename with digests attribute

Returns

True if args are valid for available digest algorithms, and False otherwise

portage.package.ebuild.fetch._check_digests(filename, digests, show_errors=1)¶: Check digests and display a message if an error occurs. :return True if all digests match, False otherwise.

portage.package.ebuild.fetch._check_distfile(filename, digests, eout, show_errors=1, hash_filter=None)¶: :return a tuple of (match, stat_obj) where match is True if filename matches all given digests (if any) and stat_obj is a stat result, or None if the file does not exist.

portage.package.ebuild.fetch._checksum_failure_temp_file(settings, distdir, basename)¶: First try to find a duplicate temp file with the same checksum and return that filename if available. Otherwise, use mkstemp to create a new unique filename._checksum_failure_.$RANDOM, rename the given file, and return the new filename. In any case, filename will be renamed or removed before this function returns a temp filename.

portage.package.ebuild.fetch._drop_privs_userfetch(settings)¶: Drop privileges for userfetch, and update portage.data.secpass to correspond to the new privilege level.

portage.package.ebuild.fetch._ensure_distdir(settings, distdir)¶

Ensure that DISTDIR exists with appropriate permissions.

Parameters

settings (portage.package.ebuild.config.config) – portage config
distdir (str) – DISTDIR path

Raises

PortageException – portage.exception wrapper exception

portage.package.ebuild.fetch._hide_url_passwd(url)¶

portage.package.ebuild.fetch._spawn_fetch(settings, args, **kwargs)¶: Spawn a process with appropriate settings for fetching, including userfetch and selinux support.

portage.package.ebuild.fetch._userpriv_test_write_file(settings, file_path)¶

Drop privileges and try to open a file for writing. The file may or may not exist, and the parent directory is assumed to exist. The file is removed before returning.

Parameters

settings – A config instance which is passed to _spawn_fetch()
file_path – A file path to open and write.

Returns

True if write succeeds, False otherwise.

portage.package.ebuild.fetch._want_userfetch(settings)¶

Check if it’s desirable to drop privileges for userfetch.

Parameters: settings (portage.package.ebuild.config.config) – portage config
Returns: True if desirable, False otherwise

portage.package.ebuild.fetch.fetch(myuris, mysettings, listonly=0, fetchonly=0, locks_in_subdir='.locks', use_locks=1, try_mirrors=1, digests=None, allow_missing_digests=True, force=False)¶

Fetch files to DISTDIR and also verify digests if they are available.

Parameters

myuris (dict) – Maps each file name to a tuple of available fetch URIs.
mysettings (portage.config) – Portage config instance.
listonly (bool) – Only print URIs and do not actually fetch them.
fetchonly (bool) – Do not block for files that are locked by a concurrent fetcher process. This means that the function can return successfully before all files have been successfully fetched!
use_locks (bool) – Enable locks. This parameter is ineffective if FEATURES=distlocks is disabled in the portage config!
digests (dict) – Maps each file name to a dict of digest types and values.
allow_missing_digests (bool) – Enable fetch even if there are no digests available for verification.
force (bool) – Force download, even when a file already exists in DISTDIR. This is most useful when there are no digests available, since otherwise download will be automatically forced if the existing file does not match the available digests. Also, this avoids the need to remove the existing file in advance, which makes it possible to atomically replace the file and avoid interference with concurrent processes.

Return type

int

Returns

1 if successful, 0 otherwise.

portage.package.ebuild.fetch.get_mirror_url(mirror_url, filename, mysettings, cache_path=None)¶

Get correct fetch URL for a given file, accounting for mirror layout configuration.

Parameters

mirror_url – Base URL to the mirror (without ‘/distfiles’)
filename – Filename to fetch
cache_path – Path for mirror metadata cache

Returns

Full URL to fetch

portage.package.ebuild.fetch module¶

Previous topic

Next topic