secretflow.utils package

Subpackages

• secretflow.utils.simulation package
  • Subpackages
  • Submodules
  • secretflow.utils.simulation.datasets module
  • secretflow.utils.simulation.tf_gnn_model module
  • Module contents

Submodules

secretflow.utils.compressor module

Classes:

Compressor()	Abstract base class for cross device data compressor
SparseCompressor(sparse_rate)
RandomSparse(sparse_rate)	Random sparse compressor compress data by randomly set element to zero.
TopkSparse(sparse_rate)	Topk sparse compressor use topK algorithm to transfer dense matrix into sparse matrix.

class secretflow.utils.compressor.Compressor

Bases: ABC

Abstract base class for cross device data compressor

Methods:

compress(data)	Compress data before send.
decompress(data)	Decompress data after receive.

abstract compress(data: Union[ndarray, List[ndarray]]) → Union[Any, List[Any]]

Compress data before send.

Parameters

data (Union[np.ndarray, List[np.ndarray]]) – data need to compress.

Returns

compressed data.

Return type

Union[Any, List[Any]]

abstract decompress(data: Union[Any, List[Any]]) → Union[ndarray, List[ndarray]]

Decompress data after receive.

Parameters

data (Union[Any, List[Any]]) – data need to decompress.

Returns

decompressed data.

Return type

Union[np.ndarray, List[np.ndarray]]

class secretflow.utils.compressor.SparseCompressor(sparse_rate: float)

Bases: Compressor

Methods:

__init__(sparse_rate)	Initialize
compress(data)	Compress data to sparse matrix before send.
decompress(data)	Decompress data from sparse matrix to dense after received.

__init__(sparse_rate: float)

Initialize

Parameters

sparse_rate – the percentage of cells are zero.

compress(data: Union[ndarray, List[ndarray]]) → Union[spmatrix, List[spmatrix]]

Compress data to sparse matrix before send.

Parameters

data (Union[np.ndarray, List[np.ndarray]]) – data need to compress.

Returns

compressed data.

Return type

Union[sparse.spmatrix, List[sparse.spmatrix]]

decompress(data: Union[spmatrix, List[spmatrix]]) → Union[ndarray, List[ndarray]]

Decompress data from sparse matrix to dense after received.

Parameters

data (Union[sparse.spmatrix, List[sparse.spmatrix]]) – data need to decompress.

Returns

decompressed data.

Return type

Union[np.ndarray, List[np.ndarray]]

class secretflow.utils.compressor.RandomSparse(sparse_rate: float)

Bases: SparseCompressor

Random sparse compressor compress data by randomly set element to zero.

Methods:

__init__(sparse_rate)

Initialize

__init__(sparse_rate: float)

Initialize

Parameters

sparse_rate – the percentage of cells are zero.

class secretflow.utils.compressor.TopkSparse(sparse_rate: float)

Bases: SparseCompressor

Topk sparse compressor use topK algorithm to transfer dense matrix into sparse matrix.

Methods:

__init__(sparse_rate)

Initialize

__init__(sparse_rate: float)

Initialize

Parameters

sparse_rate – the percentage of cells are zero.

secretflow.utils.errors module

Exceptions:

AlreadyExistsError	Raise when already exists.
InvalidArgumentError	Raise when invalid argument.
NotFoundError	Raise if not found.
PartyNotFoundError	Raise if party not found.
UnexpectedError	Raise when unexpected.
HttpNotOkError	Raise if http code is not 200

exception secretflow.utils.errors.AlreadyExistsError

Bases: Exception

Raise when already exists.

exception secretflow.utils.errors.InvalidArgumentError

Bases: Exception

Raise when invalid argument.

exception secretflow.utils.errors.NotFoundError

Bases: Exception

Raise if not found.

exception secretflow.utils.errors.PartyNotFoundError

Bases: Exception

Raise if party not found.

exception secretflow.utils.errors.UnexpectedError

Bases: Exception

Raise when unexpected.

exception secretflow.utils.errors.HttpNotOkError

Bases: Exception

Raise if http code is not 200

secretflow.utils.hash module

Functions:

sha256sum(filename)

secretflow.utils.hash.sha256sum(filename: str)

secretflow.utils.io module

Functions:

rows_count(filename)

get rows count from file

secretflow.utils.io.rows_count(filename)

get rows count from file

secretflow.utils.ndarray_bigint module

Functions:

randbits(shape, bits)
randint(shape, min, max)
arange(max)
zeros(shape)

Classes:

BigintNdArray(data, shape)

secretflow.utils.ndarray_bigint.randbits(shape: tuple, bits)

secretflow.utils.ndarray_bigint.randint(shape: tuple, min, max)

secretflow.utils.ndarray_bigint.arange(max)

secretflow.utils.ndarray_bigint.zeros(shape)

class secretflow.utils.ndarray_bigint.BigintNdArray(data, shape)

Bases: object

Methods:

__init__(data, shape)
resize(shape)
to_list()
to_numpy()
to_hnp(encoder)
to_bytes(bytes_per_int[, byteorder])

__init__(data, shape)

resize(shape)

to_list()

to_numpy()

to_hnp(encoder)

to_bytes(bytes_per_int, byteorder='little')

secretflow.utils.ndarray_encoding module

Functions:

encode(m, fxp_bits)	Encode float ndarray to uint64 finite field.
decode(m, fxp_bits)	Decode ndarray from uint64 finite field to the float.

secretflow.utils.ndarray_encoding.encode(m: ndarray, fxp_bits: int) → ndarray

Encode float ndarray to uint64 finite field. Float will times 2**fxp_bits firstly.

Parameters

• m (np.ndarray) – the ndarray to encode.

• fraction_precision (int) – keep how many decimal digits after the dot. Must provide if ndarray dtype is float.

Returns

the encoded ndarray.

Return type

np.ndarray

secretflow.utils.ndarray_encoding.decode(m: ndarray, fxp_bits: int) → ndarray

Decode ndarray from uint64 finite field to the float. Fraction precision shall be corresponding to encoding fraction precision.

Parameters

• m (np.ndarray) – the ndarray to decode.

• fxp_bits (int) – the decimal digits to keep when encoding float. Must provide if the original dtype is float.

Returns

the decoded float ndarray.

Return type

np.ndarray

secretflow.utils.sigmoid module

Functions:

t1_sig(x[, limit])	taylor series referenced from: https://mortendahl.github.io/2017/04/17/private-deep-learning-with-mpc/
t3_sig(x[, limit])	taylor series referenced from: https://mortendahl.github.io/2017/04/17/private-deep-learning-with-mpc/
t5_sig(x[, limit])	taylor series referenced from: https://mortendahl.github.io/2017/04/17/private-deep-learning-with-mpc/
seg3_sig(x)	f(x) = 0.5 + 0.125x if -4 <= x <= 4
df_sig(x)	https://dergipark.org.tr/en/download/article-file/54559 Dataflow implementation of sigmoid function: F(x) = 0.5 * ( x / ( 1 + |x| ) ) + 0.5 df_sig has higher precision than sr_sig if x in [-2, 2]
sr_sig(x)	https://en.wikipedia.org/wiki/Sigmoid_function#Examples Square Root approximation functions: F(x) = 0.5 * ( x / ( 1 + x^2 )^0.5 ) + 0.5 sr_sig almost perfect fit to sigmoid if x out of range [-3,3]
ls7_sig(x)	Polynomial fitting
mix_sig(x)	mix ls7 & sr sig, use ls7 if |x| < 4 , else use sr.
real_sig(x)
sigmoid(x, sig_type)

Classes:

SigType(value)

An enumeration.

secretflow.utils.sigmoid.t1_sig(x, limit: bool = True)

taylor series referenced from: https://mortendahl.github.io/2017/04/17/private-deep-learning-with-mpc/

secretflow.utils.sigmoid.t3_sig(x, limit: bool = True)

taylor series referenced from: https://mortendahl.github.io/2017/04/17/private-deep-learning-with-mpc/

secretflow.utils.sigmoid.t5_sig(x, limit: bool = True)

taylor series referenced from: https://mortendahl.github.io/2017/04/17/private-deep-learning-with-mpc/

secretflow.utils.sigmoid.seg3_sig(x)

f(x) = 0.5 + 0.125x if -4 <= x <= 4

1 if x > 4 0 if -4 > x

secretflow.utils.sigmoid.df_sig(x)

https://dergipark.org.tr/en/download/article-file/54559 Dataflow implementation of sigmoid function: F(x) = 0.5 * ( x / ( 1 + |x| ) ) + 0.5 df_sig has higher precision than sr_sig if x in [-2, 2]

secretflow.utils.sigmoid.sr_sig(x)

https://en.wikipedia.org/wiki/Sigmoid_function#Examples Square Root approximation functions: F(x) = 0.5 * ( x / ( 1 + x^2 )^0.5 ) + 0.5 sr_sig almost perfect fit to sigmoid if x out of range [-3,3]

secretflow.utils.sigmoid.ls7_sig(x)

Polynomial fitting

secretflow.utils.sigmoid.mix_sig(x)

mix ls7 & sr sig, use ls7 if |x| < 4 , else use sr. has higher precision in all input range. NOTICE: this method is very expensive, only use for hessian matrix.

secretflow.utils.sigmoid.real_sig(x)

class secretflow.utils.sigmoid.SigType(value)

Bases: Enum

An enumeration.

Attributes:

REAL
T1
T3
T5
DF
SR
MIX

REAL = 'real'

T1 = 't1'

T3 = 't3'

T5 = 't5'

DF = 'df'

SR = 'sr'

MIX = 'mix'

secretflow.utils.sigmoid.sigmoid(x, sig_type: SigType)

secretflow.utils.testing module

Functions:

unused_tcp_port()	Return an unused port
cluster_def(parties[, runtime_config])	Generate SPU cluster_def for testing
heu_config(sk_keeper, evaluators)

secretflow.utils.testing.unused_tcp_port() → int

Return an unused port

secretflow.utils.testing.cluster_def(parties: List[str], runtime_config=None)

Generate SPU cluster_def for testing

secretflow.utils.testing.heu_config(sk_keeper: str, evaluators: List[str])

Module contents