LoCI logo
Home
About Us
Projects
Publications
Software
Documents
Screencasts
Related Projects
Related Pubs

People
FAQs
News Archive
Contact Us


Links

IBPvo
Record your favorite show on TV!
Linux ISOs
Download the latest version of Linux!
 
 



 
Main Page   Modules   Data Structures   File List   Data Fields   Globals  

The LoRS API for Sets


Modules

The LoRS Set API Examples.

Data Structures

struct  __lorsConditionStruct
struct  __LorsSet

Functions

int lorsSetEnum (LorsSet *set, LorsEnum *list)
 Form an enumeration of all mappings within the Set.

int lorsEnumNext (LorsEnum list, LorsEnum *iterator, LorsMapping **ret)
 Iterate through a LorsEnum list of mappings.

int lorsEnumFree (LorsEnum list)
 Free a LorsEnum.

int lorsSetAddMapping (LorsSet *set, LorsMapping *map)
int lorsSetRemoveMapping (LorsSet *set, LorsMapping *map)
int lorsSetInit (LorsSet **set, ulong_t data_blocksize, int copies, int opts)
 Initialize LorsSet internal data structures.

int lorsSetStore (LorsSet *set, LorsDepotPool *dp, char *buffer, longlong offset, longlong length, LorsConditionStruct *lc, int nthreads, int timeout, int opts)
 Allocate storage and Send data to the IBP allocations.

long lorsSetLoad (LorsSet *set, char *buffer, longlong offset, long length, ulong_t block_size, LorsConditionStruct *lc, int nthreads, int timeout, int opts)
 Retrieve data from IBP allocations.

longlong lorsSetRealTimeLoad (LorsSet *set, char *buffer, int fd, longlong offset, longlong length, ulong_t block_size, int pre_buf, int cache, LorsConditionStruct *lc, int nthreads, int timeout, int max_thds_per_depot, int thds_per_job, int progress_n, int opts)
 Provide real-time streaming of data over a file descriptor.

int lorsSetCopy (LorsSet *srcSet, LorsSet *dstSet, LorsDepotPool *dp, LboneResolution *lr, longlong offset, longlong length, int nthreads, int timeout, int opts)
 Create replicas of data in srcSet allocation to another.

int lorsSetMerge (LorsSet *src, LorsSet *dest)
 Combine the mappings from two Sets into one Set.

int lorsSetTrim (LorsSet *set, longlong offset, longlong length, int nthreads, int timeout, int opts)
 Remove particular extents or Mappings from Set.

int lorsSetRefresh (LorsSet *set, longlong offset, longlong length, time_t duration, int nthreads, int timeout, int opts)
 Refresh the expiration time on allocated storage.

int lorsSetStat (LorsSet *set, int nthreads, int timeout, int opts)
 Perform a status operation on the Mappings of a Set.

int lorsSetFree (LorsSet *set, int opts)
 Perform 'free' on internal data structures.

int lorsSetTrimCaps (LorsSet *set, int opts)
 Trim IBP caps from a set of mappings.


Function Documentation

int lorsSetEnum LorsSet   set,
LorsEnum   list
 

Form an enumeration of all mappings within the Set.

In order to enumerate through all of the mapping within a Set, you may use lorsSetEnum(). It returns a LorsEnum which can then be passed to lorsEnumNext() to get each successive mapping.

Parameters:
set  A LorsSet
list  A pointer to a LorsEnum

int lorsEnumNext LorsEnum    list,
LorsEnum   iterator,
LorsMapping **    ret
 

Iterate through a LorsEnum list of mappings.

Iterate through a list of LorsMappings.

Parameters:
list  A LorsEnum list returned from lorsSetEnum().
iterator  A pointer to an iterator. Must be initially NULL.
ret  A pointer into which a LorsMapping pointer is returned.
Returns:
On list exhaustion, LORS_END is returned.

int lorsEnumFree LorsEnum    list
 

Free a LorsEnum.

Frees resources associated with a LorsEnum.

Parameters:
list 
Returns:
There are various error conditions. See list of all possible errors The LoRS Error List.

int lorsSetAddMapping LorsSet   set,
LorsMapping   map
 

Append a mapping to a given set. If the mapping is already present, and error will be returned. A mapping should exist only once per set.

Parameters:
set 
map 
Returns:
There are various error conditions. See list of all possible errors The LoRS Error List.

int lorsSetRemoveMapping LorsSet   set,
LorsMapping   map
 

Remove a mapping from a given set.

Parameters:
set 
map 
Returns:
There are various error conditions. See list of all possible errors The LoRS Error List.

int lorsSetInit LorsSet **    set,
ulong_t    data_blocksize,
int    copies,
int    opts
 

Initialize LorsSet internal data structures.

lorsSetInit() initializes a LorsSet structure with the specified data_blocksize and number of copies. Before calling either lorsSetStore() or lorsSetCopy() you should call lorsSetInit() to create a target LorsSet into which new mappings will be added.

Parameters:
set  A pointer to a pointer of a LorsSet into which the new structure will be returned.
data_blocksize  The blocksize of allocated fragments.
copies  The number of replicas to create.
opts  This is always zero for now. In the future there may be more options.
Returns:
There are various error conditions. See list of all possible errors The LoRS Error List.

int lorsSetStore LorsSet   set,
LorsDepotPool   dp,
char *    buffer,
longlong    offset,
longlong    length,
LorsConditionStruct   lc,
int    nthreads,
int    timeout,
int    opts
 

Allocate storage and Send data to the IBP allocations.

lorsSetStore() sends data to IBP allocations. The number of threads used to execute the operation will depend heavily on the initial blocksize given to lorsSetCreate() and the size of buffer. Only one thread can be active per Mapping (i.e. per replica).
In the event of depot failure during Store or Allocate, a new buffer will be allocated from the LorsDepotPool.

Parameters:
set  A LorsSet from a previous call to lorsSetCreate() or a deserialize operation.
dp  A LorsDepotPool from which new allocations will be drawn.
buffer  The data a user wishes to upload.
offset  The absolute offset relative to the exnode to store the buffer data. This allows multiple calls to SetStore simultaneously, if necessary.
length  The length of buffer.
lc  An array of LorsConditionStructs which specify the order to apply end2end operations. This may be NULL if no conditioning is desired.
nthreads  The maximum number of threads to use. By default, only (length/blocksize + 1 * copies) can be used,
timeout  Return after timeout seconds, even if operation is incomplete.
opts 
  • LORS_RETURN_ON_ANY_ERROR
  • LORS_RETRY_UNTIL_TIMEOUT
Returns:
There are various error conditions. See list of all possible errors The LoRS Error List.

long lorsSetLoad LorsSet   set,
char *    buffer,
longlong    offset,
long    length,
ulong_t    block_size,
LorsConditionStruct   lc,
int    nthreads,
int    timeout,
int    opts
 

Retrieve data from IBP allocations.

lorsSetLoad() retrieves data from IBP and stores it into buffer. buffer must contain at least length bytes, or risk overwriting memory.
The number of threads used depends strictly on the amount passed to nthreads and the opts Policy.
lorsSetLoad() operates on 'length/nthreads' Blocksizes. However, in the case where 'Blocksize' is larger than the length to next boundary point (i.e. when mappings size are not an even multiple of 'Blocksize') the less of the two will be chosen to assign the next thread.
The return value is the number of bytes read.

Parameters:
set  A LorsSet from a previous call to lorsSetCreate() or a deserialize operation.
buffer  A pointer to data area into which lorsSetLoad() stores.
offset  The offset relative to the Set from which this operation is to begin.
length  The desired size of set to retrieve, and length of buffer
block_size  The maximum datasize for one IBP_load call.
lc  An array of all possible LorsConditionStructs used to decode any conditioning applied to the mappings in set.
nthreads  The maximum number of threads to use. Because IBP allocations are memory mapped, any number of simultaneous threads may be used.
timeout  Return after timeout seconds, even if operation is incomplete.
opts 
  • LORS_RETURN_ON_ANY_ERROR
  • LORS_RETRY_UNTIL_TIMEOUT
Returns:
On error, a negative number is returned. Otherwise the number of bytes successfully written to the buffer is returned.

longlong lorsSetRealTimeLoad LorsSet   set,
char *    buffer,
int    fd,
longlong    offset,
longlong    length,
ulong_t    block_size,
int    pre_buf,
int    cache,
LorsConditionStruct   lc,
int    nthreads,
int    timeout,
int    max_thds_per_depot,
int    thds_per_job,
int    progress_n,
int    opts
 

Provide real-time streaming of data over a file descriptor.

To satisfy the requirements of real time streaming applications, lorsSetRealTimeLoad() writes to a file descriptor rather than a buffer. Most of the semantics remain the same as for lorsSetLoad().

Parameters:
set  A LorsSet from a previous call to lorsSetCreate() or a deserialize operation.
buffer  A pointer to data area into which lorsSetLoad() stores.
fd  A valid file descriptor to which lorsSetRealTimeLoad() will write in order the successfully downloaded length requested.
offset  The offset relative to the Set from which this operation is to begin.
length  The desired size of set to retrieve, and length of buffer
block_size  The maximum datasize for one IBP_load call.
pre_buf  This is an integer count of the number of block_size sections to fully download before writing to the file descriptor.
cache  The maximum amount of buffering the call will provide in addition to the memory allocated per-thread.
lc  An array of all possible LorsConditionStructs used to decode any conditioning applied to the mappings in set.
nthreads  The maximum number of threads to use. Because IBP allocations are memory mapped, any number of simultaneous threads may be used.
timeout  Return after timeout seconds, even if operation is incomplete.
max_thds_per_depot  In order to minimize the impact of our algorithm on depots, this parameter will provide a cap for individual depot connections.
thds_per_job  This is a measure of redundancy.
progress_n  When progress_n is greater than 0, and thds_per_job is greater than 1
opts 
  • LORS_RETURN_ON_ANY_ERROR
  • LORS_RETRY_UNTIL_TIMEOUT
Returns:
On error, a negative number is returned. Otherwise the number of bytes successfully written to the buffer is returned.

int lorsSetCopy LorsSet   srcSet,
LorsSet   dstSet,
LorsDepotPool   dp,
LboneResolution   lr,
longlong    offset,
longlong    length,
int    nthreads,
int    timeout,
int    opts
 

Create replicas of data in srcSet allocation to another.

lorsSetCopy() copies data from the @offset and length of srcSet to the seek_offset of dstSet. If the length of of srcSet is too large to fit within dstSet, it may or may not generate an error condition depending on the options passed in opts.

Parameters:
srcSet  A set from which a copy begins.
dstSet  A pointer to a set to which the copy is placed.
dp  A LorsDepotPool from which new allocations will be drawn.
lr  An LboneResolution matrix as returned by lorsCreateResolutionFromFile() or others. From this information, decision regarding routing are made.
offset  The exnode offset of the mappings in srcSet
length  Length relative to srcSet
nthreads  By default, this will depend on the number of mappings in set , i.e. one thread per fragment.
timeout  Return after timeout seconds, even if operation is incomplete.
opts 
  • LORS_RETURN_ON_ANY_ERROR
  • LORS_RETRY_UNTIL_TIMEOUT
  • LORS_MCOPY
  • LORS_COPY
Returns:
There are various error conditions. See list of all possible errors The LoRS Error List.

int lorsSetMerge LorsSet   src,
LorsSet   dest
 

Combine the mappings from two Sets into one Set.

lorsSetMerge() combines two Sets into a single dest, It does not perform a deep copy on internal datastructures, and mappings. The two are not independent after lorsSetMerge().

Parameters:
src  The pointer to the first and source Set.
dest  The pointer to second and destination Set.
Returns:
There are various error conditions. See list of all possible errors The LoRS Error List.

int lorsSetTrim LorsSet   set,
longlong    offset,
longlong    length,
int    nthreads,
int    timeout,
int    opts
 

Remove particular extents or Mappings from Set.

After an allocation has expired it may be of no use to leave it in the Set. lorsSetTrim() provides the mechanism by which expired or unneeded mappings may be removed. Alternately, the operation is more subtle, allowing for readdressing of existing mappings, such that, pieces of a mapping may be ignored, while still present, in IBP.
The number of threads needed to perform this operation are determined automatically.

Parameters:
set  A LorsSet from a previous call to lorsSetCreate() or a deserialize operation.
offset  The offset relative to the Set.
length  The desired size of set to trim.
nthreads  By default, this will depend on the number of mappings in set , i.e. one thread per fragment.
timeout  Return after timeout seconds, even if operation is incomplete.
opts 
  • LORS_TRIM_ALL -
  • LORS_TRIM_DEAD -
  • LORS_TRIM_KILL -
  • LORS_TRIM_NOKILL -
  • LORS_RETURN_ON_ANY_ERROR
  • LORS_RETRY_UNTIL_TIMEOUT
Returns:
There are various error conditions. See list of all possible errors The LoRS Error List.

int lorsSetRefresh LorsSet   set,
longlong    offset,
longlong    length,
time_t    duration,
int    nthreads,
int    timeout,
int    opts
 

Refresh the expiration time on allocated storage.

Because the underlying storage of the LoRS tools is IBP, the allocations will eventually expire. However, before they expire, it may be possible to renew the lease on these allocations.
The number of threads are determined automatically if 'nthreads' is -1.

Parameters:
set  A LorsSet from a previous call to lorsSetCreate() or a deserialize operation.
offset  The offset relative to the Set.
length  The desired size of set to refresh.
duration  The new time in seconds.
nthreads  By default, this will depend on the number of mappings in set , i.e. one thread per fragment.
timeout  Return after timeout seconds, even if operation is incomplete.
opts 
  • LORS_REFRESH_MAX - Attempt to refresh to the largest allowable expiration time.
  • LORS_REFRESH_EXTEND_BY - Attempt to add duration seconds to the time out.
  • LORS_REFRESH_EXTEND_TO - Attempt to normalize all duration seconds from now.
  • LORS_REFRESH_ABSOLUTE - Attempt to set expiration to duration seconds.
  • LORS_RETURN_ON_ANY_ERROR
  • LORS_RETRY_UNTIL_TIMEOUT
Returns:
There are various error conditions. See list of all possible errors The LoRS Error List.

int lorsSetStat LorsSet   set,
int    nthreads,
int    timeout,
int    opts
 

Perform a status operation on the Mappings of a Set.

In order to have some view of the Mapping level, lorsSetStat() is provided to probe the data of individaul allocations. This allows for more complex operations such as 'Balance' which could guarantee a minimum number of replicas per-offset, or for determining other reliability statistics.

Parameters:
set  A LorsSet from a previous call to lorsSetCreate() or a deserialize operation.
nthreads  The maximum number of threads to use for this operation. a value of -1 will use 1 thread per mapping.
timeout  The maximum amount of time to allow for the entire operation, regardless of success or failure.
opts 
  • LORS_STAT_LIVE
  • LORS_STAT_CACHE
  • LORS_RETURN_ON_ANY_ERROR
  • LORS_RETRY_UNTIL_TIMEOUT
Returns:
There are various error conditions. See list of all possible errors The LoRS Error List.

int lorsSetFree LorsSet   set,
int    opts
 

Perform 'free' on internal data structures.

Release memory back to local system.

Parameters:
set  A LorsSet
opts 
  • LORS_FREE_MAPPINGS
Returns:
0

int lorsSetTrimCaps LorsSet   set,
int    opts
 

Trim IBP caps from a set of mappings.

Trim IBP caps from a set of mappings.

Parameters:
set  A LorsSet
opts 
  • LORS_READ_CAP
  • LORS_WRITE_CAP
  • LORS_MANAGE_CAP
Returns:
There are various error conditions. See list of all possible errors The LoRS Error List.

UTK home        DOE NSF
 
corner
Home   About Us   Projects   Publications   Software   Docs   Screencasts   Related Projects   Related Pubs   People   FAQs   News  Contact Us
corner