File Management

Suggest edits
Documentation > Language

Contents

OpenMOLE implements mechanisms to conveniently make use of data stored as files in your experiments.
Nota Bene: OpenMOLE makes very few distinctions between files and directories. In this part, most mentions of a "file" are also valid for a "directory".

The files in OpenMOLE tasks have two main use cases:

Input files and resources ๐Ÿ”—

A task can be provided with input files and resources prior to its execution, in order to use thses files during execution. The difference between input files and resources is that input files are generally produced dynamically by another task in the workflow, whereas resources are preexisting files.
In OpenMOLE, there are two behaviors when it comes to manipulating input files and resources: one for the ScalaTask, and one for the other "external" tasks that are mainly used to embed code from other languages.

External Tasks ๐Ÿ”—

In order to provide files to that kind of tasks (Python, R, Netlogo, ...), OpenMOLEย needs to copy the file to a known path on the local hard drive and feed it to the task during execution.
Resources
To provide a file as a resource to a task, it first needs to be uploaded to your working directory in the GUI (see here for more info on how to do this). It can then be fed to the task through the resources keyword inside the task: resources += path/to/the/file.
Let's first consider a simple case in which an external task requires a file named file.txt to be executed, always the same:

val task = SystemExecTask("cat file.txt") set (
   resources += workDirectory / "file.txt"
)

task

We can do the same with a directory present in your OpenMOLE working directory:

val task = SystemExecTask("ls mydirectory") set (
  resources += workDirectory / "mydirectory"
)

task

It is also possible to provide a second argument to rename the file, in this way: resources += (path/to/the/file, "new_name.txt"). In this case the directory in which the task is executed contains a file with the same content but a different name. For instance:

val task = SystemExecTask("cat bettername.txt") set (
  resources += (workDirectory / "file.txt", "bettername.txt")
)

task

The resources keyword is useful for files existing before the workflow execution. In other cases, you might want to produce a file in a task and provide it to a subsequent task.
Input files
Input files are files that the workflow will interact dynamically with, mainly by creating them, or writing in them. The inputFiles keyword is used for these files, assigning a file variable from the workflow to a name and linking it to the file object created: inputFiles += (fileVariable, "filename.txt").

// Create the file variable in the workflow
val f = Val[File]

// Create the file and write something in it
val producer = ScalaTask("""
    val f = newFile()
    f.content = "I love tofu"
""") set (
    outputs += f
)

// Take file f, name it love.txt, and display it
val task = SystemExecTask("cat love.txt") set (
    inputFiles += (f, "love.txt")
)

// Chain the two tasks
producer -- task

Scala Tasks ๐Ÿ”—

Since the ScalaTask is able to directly access the file variable Val[File], it is easier to provide files to a ScalaTask. Files are just plain simple inputs, in the same manner as any other inputs.

val f = Val[File]

val task = ScalaTask("""
  println(f.content)
""") set (
  inputs += f,
  // default value
  f := workDirectory / "file.txt"
)

task

In the example above, the file is provided using a default argument. You can of course produce it in another task.

val f = Val[File]

val producer = ScalaTask("""
  val f = newFile()
  f.content = "I love tofu"
""") set (
  outputs += f
)

val task = ScalaTask("""
  println(f.content)
""") set (
  inputs += f
)

producer -- task

Note that the type of variable f is java.io.File. It can be provided as an argument to Java or Scala function calls as it is.

Output files ๐Ÿ”—

When files are produced by a task, they should be set as an output of this task in order to be collected and persist after the execution of the task. These files are assigned to a variable of type Val[File] and can be transmitted to another task, or copied in the work directory using CopyFileHook (see here for more info on hooks). For external tasks (all but the ScalaTask), the way to collect files after a task execution is by using the outputFiles keyword. The case of the ScalaTask is explained in a separate section.

External Tasks ๐Ÿ”—

The general mechanism to save a file for subsequent execution or in the work directory is to link the path of the produced file to a variable of type Val[File] using the outputFiles keyword. outputFiles gets a file with a given name and assign it to an OpenMOLE variable once the execution of the task is completed: outputFiles += ("filename.txt", fileVariable).

// Create the file variable f
val f = Val[File]

// Produce the file and save it in the f object
val task = SystemExecTask("echo I love tofu > output.txt") set (
    outputFiles += ("output.txt", f)
)

// Hook the task to write f in the working directory
task hook CopyFileHook(f, workDirectory / "taskOutput.txt")

Scala Task ๐Ÿ”—

When using the ScalaTask, files are simple variables and are manipulated like any variable. To output a file variable you can just set it as any usual output:

val f = Val[File]

val producer = ScalaTask("""
  val f = newFile()
  f.content = "I love tofu"
""") set (
  outputs += f
)

Complementary information ๐Ÿ”—