sirius.biz.jdbc.storage

Class Storage

java.lang.Object

sirius.biz.jdbc.storage.Storage

public class Storage
extends Object

Provides the main facility for storing and retrieving objects (files) to or from buckets.

The object storage is organized in two layers. The virtual layer is completely stored in the database and contains all metadata of buckets and objects.

Everytime an object is created or updated, a physical object is created and stored using the PhysicalStorageEngine of the bucket. Therefore a physical object key which is stored in the virtual objects always contains the same data. If the data changes, the physical key changes but the virtual key remains the same.

Using this approach, the generated URLs can be cached without having the worry about this in the data model.

For database entities referencing virtual objects a StoredObjectRef can be used, which takes care of referential integrity (deletes the object, if the entity is deleted etc.)

Field Summary

Fields

Modifier and Type	Field and Description
static String	FRAMEWORK_STORAGE Names the framework which must be enabled to activate the storage feature.
static sirius.kernel.health.Log	LOG Logger used by the storage facility.

Constructor Summary

Constructors

Constructor and Description
Storage()

Method Summary

Modifier and Type	Method and Description
String	calculateMd5(File file) Computes a base64 representation of the md5 sum of the given file.
protected void	clearCacheForVirtualObject(VirtualObject virtualObject) Clears the cache for a given VirtualObject.
StoredObject	createTemporaryObject(Tenant tenant, String bucketName, String reference, String path) Creates a new temporary object to be used in a StoredObjectRef.
protected String	createURL(DownloadBuilder downloadBuilder) Creates a download URL for a fully populated builder.
void	delete(StoredObject object) Deletes the given object and alls of its versions.
protected void	deletePhysicalObject(String bucket, String physicalKey) Deletes the physical object in the given bucket.
protected void	deleteReferencedObjects(String reference, String excludedObjectKey) Deletes all automatically created objects except the given one.
protected void	deliverPhysicalFile(sirius.web.http.WebContext ctx, String bucket, String physicalKey, String fileExtension) Delivers a pyhsical file or object.
Optional<StoredObject>	findByKey(Tenant tenant, String bucket, String key) Tries to find the object with the given id, for the given tenant and bucket.
Optional<StoredObject>	findByPath(Tenant tenant, String bucketName, String path) Tries to find the object with the given path, for the given tenant and bucket.
StoredObject	findOrCreateObjectByPath(Tenant tenant, String bucketName, String path) Tries to find the object with the given path, for the given tenant and bucket.
Optional<BucketInfo>	getBucket(String bucket) Returns the bucket infos for the bucket with the given name.
Collection<BucketInfo>	getBuckets() Returns a list of all known buckets known to the system.
InputStream	getData(StoredObject file) Retrieves the actual data stored by the given object.
protected PhysicalStorageEngine	getStorageEngine(String bucketName)
void	list(BucketInfo bucket, Tenant tenant, Function<StoredObject,Boolean> iterator) Lists all objects within the given bucket for the given tenant.
protected void	markAsUsed(String reference, String objectKey) If an object is created using createTemporaryObject(Tenant, String, String, String) it is marked as an temporary upload up until the referencing entity is saved.
static String	normalizePath(String path) Normalizes the given path to be used in VirtualObject.PATH
DownloadBuilder	prepareDownload(String bucket, String objectKey) Creates a builder to construct a download URL for an object.
protected DownloadBuilder	prepareDownload(VirtualObject object) Creates a builder for download URLs based on a VirtualObject which might avoid a lookup.
OutputStream	updateFile(StoredObject file) Creates a new output stream which updates the contents of the given file.
void	updateFile(StoredObject file, File data, String filename) Updates the given file using the given data.
void	updateFile(StoredObject file, InputStream data, String filename, String md5, long size) Updates the given file based on the given stream.
protected boolean	verifyHash(String key, String hash) Verifies the authentication hash for the given key.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final sirius.kernel.health.Log LOG

Logger used by the storage facility.

FRAMEWORK_STORAGE

public static final String FRAMEWORK_STORAGE

Names the framework which must be enabled to activate the storage feature.

See Also:

Constant Field Values

Constructor Detail

Storage

public Storage()

Method Detail

getBuckets

public Collection<BucketInfo> getBuckets()

Returns a list of all known buckets known to the system.

Returns:

a collection of all buckets known to the system (declared in storage.buckets in the system config).

getBucket

public Optional<BucketInfo> getBucket(String bucket)

Returns the bucket infos for the bucket with the given name.

Parameters:

bucket - the name of the bucket to fetch infos for

Returns:

the bucket wrapped as optional or an empty optional if the bucket is unknown

getStorageEngine

protected PhysicalStorageEngine getStorageEngine(String bucketName)

findByKey

public Optional<StoredObject> findByKey(@Nullable
                                        Tenant tenant,
                                        String bucket,
                                        String key)

Tries to find the object with the given id, for the given tenant and bucket.

Uses a cache for the virtual objects. When fetching an object from the cache, the buckets and tennants are compared to ensure integrity. Only non-temporary objects are cached.

Parameters:

tenant - the tenant to filter on

bucket - the bucket to search in

key - the globally unique id to search by

Returns:

the object in the given bucket with the given key wrapped as optional or an empty optional if no such object exists.

clearCacheForVirtualObject

protected void clearCacheForVirtualObject(VirtualObject virtualObject)

Clears the cache for a given VirtualObject.

Parameters:

virtualObject - the virtual object

list

public void list(@Nullable
                 BucketInfo bucket,
                 @Nullable
                 Tenant tenant,
                 Function<StoredObject,Boolean> iterator)

Lists all objects within the given bucket for the given tenant.

Parameters:

bucket - the bucket to search in

tenant - the tenant to filter on

iterator - the iterator to process the objects

normalizePath

public static String normalizePath(String path)

Normalizes the given path to be used in VirtualObject.PATH

Parameters:

path - the path to cleanup

Returns:

the normalized path without \ or // or " "

findByPath

public Optional<StoredObject> findByPath(Tenant tenant,
                                         String bucketName,
                                         String path)

Tries to find the object with the given path, for the given tenant and bucket.

Parameters:

tenant - the tenant to filter on

bucketName - the bucket to search in

path - the path used to lookup the object

Returns:

the object in the given bucket with the given path wrapped as optional or an empty optional if no such object exists.

findOrCreateObjectByPath

public StoredObject findOrCreateObjectByPath(Tenant tenant,
                                             String bucketName,
                                             String path)

Tries to find the object with the given path, for the given tenant and bucket. If it doesn't exist, it is created.

Parameters:

tenant - the tenant to filter on

bucketName - the bucket to search in

path - the path used to lookup the object

Returns:

the object in the given bucket with the given path which was either found or newly created

createTemporaryObject

public StoredObject createTemporaryObject(Tenant tenant,
                                          String bucketName,
                                          @Nullable
                                          String reference,
                                          @Nullable
                                          String path)

Creates a new temporary object to be used in a StoredObjectRef.

Such objects will be invisible to the user and also be automatically deleted if the referencing entity is deleted. It is made permanent as soon as the referencing entity is saved.

Parameters:

tenant - the tenant owning the object

bucketName - the bucket in which the object is placed

reference - the reference for which the object was created

path - the path of the object to be created

Returns:

the newly created object which has VirtualObject.TEMPORARY set to true. Therefore the referencing entity must be saved to set this flag to false via StoredObjectRefProperty.

deleteReferencedObjects

protected void deleteReferencedObjects(String reference,
                                       @Nullable
                                       String excludedObjectKey)

Deletes all automatically created objects except the given one.

Removes automatically created object (createTemporaryObject(Tenant, String, String, String)) if the reference is updated or the referencing entity is deleted.

Parameters:

reference - the reference (field+id of the entity) being referenced

excludedObjectKey - if the reference is just changed, the new object is excluded, to just delete the old objects

markAsUsed

protected void markAsUsed(String reference,
                          String objectKey)

If an object is created using createTemporaryObject(Tenant, String, String, String) it is marked as an temporary upload up until the referencing entity is saved.

The StoredObjectRef and its StoredObjectRefProperty will then invoke this method to make the object permanently stored.

Parameters:

reference - the reference to attach to the object if it was newly created

objectKey - the object to mark as permanent

calculateMd5

public String calculateMd5(File file)
                    throws IOException

Computes a base64 representation of the md5 sum of the given file.

Parameters:

file - the file to hash

Returns:

a base64 encoded string representing the md5 hash of the given file

Throws:

IOException - in case of an IO error

updateFile

public void updateFile(@Nonnull
                       StoredObject file,
                       @Nonnull
                       File data,
                       @Nullable
                       String filename)

Updates the given file using the given data.

Parameters:

file - the S3 file to update

data - the data to store

filename - the original filename to save

updateFile

public void updateFile(@Nonnull
                       StoredObject file,
                       @Nonnull
                       InputStream data,
                       @Nullable
                       String filename,
                       @Nullable
                       String md5,
                       long size)

Updates the given file based on the given stream.

Parameters:

file - the S3 file to update

data - the data to store

filename - the original filename to save

md5 - the md5 hash value to use to verify the upload

size - the size (number of bytes) being uploaded (must be positive)

updateFile

public OutputStream updateFile(@Nonnull
                               StoredObject file)

Creates a new output stream which updates the contents of the given file.

Note that most probably, the file will be updated once the stream is closed and not immediatelly on a write. Also note that it is essential to close the stream to release underlying resources.

Parameters:

file - the file to update.

Returns:

an output stream to write the contents to.

getData

@Nonnull
public InputStream getData(@Nonnull
                                    StoredObject file)

Retrieves the actual data stored by the given object.

Parameters:

file - the object to fetch the contents for

Returns:

an input stream which provides the contents of the object / file

delete

public void delete(StoredObject object)

Deletes the given object and alls of its versions.

Parameters:

object - the object to delete

deletePhysicalObject

protected void deletePhysicalObject(String bucket,
                                    String physicalKey)

Deletes the physical object in the given bucket.

Parameters:

bucket - the bucket to delete the object from

physicalKey - the physical key of the object to delete

verifyHash

protected boolean verifyHash(String key,
                             String hash)

Verifies the authentication hash for the given key.

Parameters:

key - the key to verify

hash - the hash to verify

Returns:

true if the hash verifies the given object key, false otherwise

deliverPhysicalFile

protected void deliverPhysicalFile(sirius.web.http.WebContext ctx,
                                   String bucket,
                                   String physicalKey,
                                   String fileExtension)

Delivers a pyhsical file or object.

Parameters:

ctx - the request to respond to

bucket - the bucket to deliver from

physicalKey - the physical file to deliver

fileExtension - the file extension of the file (to determine the Content-Type)

prepareDownload

public DownloadBuilder prepareDownload(String bucket,
                                       String objectKey)

Creates a builder to construct a download URL for an object.

Parameters:

bucket - the bucket containing the object

objectKey - the object to create an URL for

Returns:

a builder to construct a download URL

prepareDownload

protected DownloadBuilder prepareDownload(VirtualObject object)

Creates a builder for download URLs based on a VirtualObject which might avoid a lookup.

Parameters:

object - the object to deliver

Returns:

a builder to construct a download URL

createURL

protected String createURL(DownloadBuilder downloadBuilder)

Creates a download URL for a fully populated builder.

Parameters:

downloadBuilder - the builder specifying the details of the url

Returns:

a download URL for the object described by the builder