File Transfer

File transfer enables sharing of large data blobs across the network.

The FileTransfer service

Python

from hyveos_sdk import Connection
import asyncio


async with Connection() as connection:
    file_transfer = connection.get_file_transfer_service()

    # continue with usage of file_transfer
    # TODO

Rust

use hyveos_sdk::Connection;


#[tokio::main]
async fn main() {
    let connection = Connection::new().await.unwrap();
    let mut file_transfer = connection.file_transfer();

    // continue with usage of file_transfer
    // todo!()
}

Typescript/JS

import { Client } from 'hyveos-sdk'
import { Connection } from 'hyveos-web'


async function main() {
    const transport = new Connection('http://localhost:8080')
    const client = new Client(transport)
    const filetransferService = client.filetransfer

    // continue with usage of filetransferService
    // TODO
}

Example 1. Obtain the File Transfer service handler. Note the async environment.

Publishing a File

A file can be uploaded from the local filesystem by using the publish_file interface.

Internally, all files are stored into a local directory, set by the HYVEOS_BRIDGE_SHARED_DIR, and are transmitted directly to requesting nodes using request-response.

Python

publishing_node.py

from hyveos_sdk import Connection
import os
import asyncio
import aiofiles
from pathlib import Path


async def main():
    async with Connection() as connection:
        file_transfer = connection.get_file_transfer_service()

        shared_dir = os.environ["HYVEOS_BRIDGE_SHARED_DIR"]

        file_path = Path(shared_dir) / "example.txt"

        async with aiofiles.open(file_path, "w") as f:
            await f.write("Hello, world!")

        cid = await file_transfer.publish_file(file_path)

        print(f"Content ID: {cid}")

asyncio.run(main())

Rust

publishing_node.rs

use hyveos_sdk::Connection;
use std::path::Path;


#[tokio::main]
async fn main() {
    let connection = Connection::new().await.unwrap();
    let mut file_transfer = connection.file_transfer();

    let shared_dir = std::env::var("HYVEOS_BRIDGE_SHARED_DIR").unwrap();
    let file_path = Path::new(&shared_dir).join("example.txt");

    tokio::fs::write(&file_path, "Hello, world!").await.unwrap();

    let cid = file_transfer.publish_file(&file_path).await.unwrap();

    println!("Content ID: {cid:?}");
}

Typescript/JS

publishing_node.ts

import { Client } from 'hyveos-sdk'
import { Connection } from 'hyveos-web'


async function main() {
    const transport = new Connection('http://localhost:8080')
    const client = new Client(transport)
    const filetransferService = client.filetransfer

    const sharedDir = process.env["HYVEOS_BRIDGE_SHARED_DIR"]

    const filePath = path.join(sharedDir, 'example.txt')

    await fs.writeFile(filePath, 'Hello, world!')

    const cid = await fileTransfer.publishFile(filePath)

    console.log(`Content ID: ${cid}`)
}

main().catch(err => { console.error('Error running main:', err) process.exit(1) })

Example 2. Publish example.txt into a shared directory on the hyve. The path to the directory is saved under the environment variable HYVEOS_BRIDGE_SHARED_DIR per default. Write "Hello, world!" into example.txt, and finally publish the file to the hyve shared directory.

File Token Content ID

Observe that the publish interface returns a Content ID (cid), which is a hash of the uploaded file. It will be used by other peers to download the file. The cid can then be advertised to other nodes by using for example the DHT service

Downloading a File to a Node

The get_file interface lets nodes download files, previously uploaded into the network, by their cid The method returns the file’s path.

When the local runtime doesn’t own a copy of this file yet, it downloads it directly from the uploading node.

The file is copied into the shared directory,which is defined by the HYVEOS_BRIDGE_SHARED_DIR environment variable.

Python

downloading_node.py

import asyncio
from hyveos_sdk import Connection


async def main():
    async with Connection() as connection:
        dht_service = connection.get_dht_service()
        cid = await dht_service.get_record_json("file", "example")

        if cid:
            file_transfer = connection.get_file_transfer_service()
            path = await file_tranfser.get_file(cid)

            print(f"File path: {path}")

            async with aiofiles.open(path, "r") as f:
                contents = await f.read()

            print(f"File length: {len(contents)}")
        else:
            print("File not found")

asyncio.run(main())

Rust

downloading_node.rs

use hyveos_sdk::Connection;


#[tokio::main]
async fn main() {
    let connection = Connection::new().await.unwrap();
    let mut dht_service = connection.dht();
    let cid = dht_service.get_record_json("file", "example").await.unwrap();

    if let Some(cid) = cid {
        let mut file_transfer = connection.file_transfer();

        let path = file_transfer.get_file(cid).await.unwrap();

        println!("File path: {}", path.display());

        let contents = tokio::fs::read_to_string(path).await.unwrap();
        println!("File length: {}", contents.len());
    } else {
        println!("File not found");
    }
}

Typescript/JS

downloading_node.ts

import { Client } from 'hyveos-sdk'
import { Connection } from 'hyveos-web'


async function main() {
    const transport = new Connection('http://localhost:8080')
    const client = new Client(transport)
    const dhtService = client.dht
    const cid = await dhtService.getRecordJson("file", "example")

    if (cid) {
        const fileTransferService = connection.filetransfer

        const stream = await fileTransferService.getFile(cid)
        // stream: ReadableStream
        // need to read out the stream to get the file data
        const reader = pathOrStream.getReader()
        while (true) {
            const { done, value } = await reader.read()
            if (done) break;
            totalSize += value?.length || 0
        }
        console.log('File length: ${totalSize}')
    } else {
        console.log('File not found')
    }
}

main().catch(err => { console.error('Error running main:', err) process.exit(1) })

Example 3. Retrieve a file from the hyve network. Before retrieval, the cid was obtained from the DHT.

Rust: Download Progress

Rust Add-on: Download Progress

The Rust SDK offers the possibility to download files with progress updates, by the get_file_with_progress interface. While downloading, it emits events with the download progress as a percentage.

use indicatif::ProgressBar;

    ...

    let mut stream = file_transfer_service
        .get_file_with_progress(cid)
        .await
        .unwrap();

    let progress_bar = ProgressBar::new(100);

    let path = loop {
        match stream.next().await.unwrap().unwrap() {
            DownloadEvent::Progress(progress) => {
                progress_bar.set_position(progress);
            }
            DownloadEvent::Ready(path) => {
                progress_bar.finish();
                break path;
            }
        }