File Management

Suggest edits
Documentation > Language

Contents

OpenMOLE implements mechanisms to conveniently make use of data stored as file in your experiments. OpenMOLE make very few distinction between file and directories. In this part when we use the "file" word it can generally be replaced by the "directory" word and the sentence hold.
The files in OpenMOLE tasks have 2 main use cases:
  • they can be provided to a task (cf. input files, resources),
  • they can be produced by a task (cf. output files).


Input files and resources ๐Ÿ”—

Input files and resources are used to provide files to a task prior to its execution. It may then make use of it when the task is executed. The difference between inputs files and resources is that inputs files are generally produced dynamically by another task in the workflow, whereas resources are used for preexisting files.
Considering inputs files and resources, the ScalaTask and the other tasks have very different behaviour. Despite the other tasks, the Scala task is able to make use of Val[File] directly. Let's first talk about the general case non-ScalaTask or external tasks.

External Tasks ๐Ÿ”—

In order to provide files to that kind of tasks (phython, R, Netlogo...) OpenMOLEย needs to copy (link) the file to a known path on the local hard drive in order for the task to open it when it runs. Let's first consider a simple case in which an external task require a file to be executed, always the same. In this case we use the resource keyword.
val task = SystemExecTask("cat file.txt") set (
  resources += workDirectory / "file.txt"
)

task
In this case the ls process is executed by openmole in a directory where the only file is "file.txt". We can do the same with a directory.
val task = SystemExecTask("ls mydirectory") set (
  resources += workDirectory / "mydirectory"
)

task
You can provide a second argument to rename the file. In this case the directory in which the task is executed contains a file with the same content but a different name. For instance:
val task = SystemExecTask("cat bettername.txt") set (
  resources += (workDirectory / "file.txt", "bettername.txt")
)

task
The resources keyword is useful for files, existing before the workflow execution. In other cases you might want to produce a file in a task and provide it to a subsequent task. For this use the inputFiles keyword.
val f = Val[File]

val producer = ScalaTask("""
  val f = newFile()
  f.content = "I love tofu"
""") set (
  outputs += f
)

val task = SystemExecTask("cat love.txt") set (
  inputFiles += (f, "love.txt")
)

producer -- task

Scala Tasks ๐Ÿ”—

Since the ScalaTask is able to directly access the file variable, it is easier to provide files to a ScalaTask. Files are just plain simple inputs, in the same manner as the other inputs.
val f = Val[File]

val task = ScalaTask("""
  println(f.content)
""") set (
  inputs += f,
  f := workDirectory / "file.txt"
)

task
In this case the file is provided using a default argument. You can of course produce it in another task.
val f = Val[File]

val producer = ScalaTask("""
  val f = newFile()
  f.content = "I love tofu"
""") set (
  outputs += f
)

val task = ScalaTask("""
  println(f.content)
""") set (
  inputs += f
)

producer -- task
Note that the type of variable f is java.io.File. It can be provided as an argument to java or scala function calls as it is.

Output files ๐Ÿ”—

When files are produced by a task they should be set as output of task in order to be collected and persist longer than the task execution time lapse. These files are assigned to a variable of type Val[File] and can be transmitted to another task or copied in the work directory using CopyFileHook. For external tasks (all but the ScalaTask), the manner to collect files after a task execution is by using the outputFiles keyword. The case of the ScalaTask is explain in a separate section.

External Tasks ๐Ÿ”—

When a task produces a file it has to be set as an output of the task in order to be saved in your OpenMOLE workDirectory and/or to be used by a subsequent task. The general mechanism is to link the path of produced file to a variable of type Val[File] using the outputFiles keyword.
val f = Val[File]

val task = SystemExecTask("echo I love tofu >output.txt") set (
  outputFiles += ("output.txt", f)
)

task hook CopyFileHook(f, workDirectory / "taskOutput.txt")
outputFiles get a file with a given name and assign it to an OpenMOLE variable once the execution of the task is completed.

Scala Task ๐Ÿ”—

When using the ScalaTask, files are simple variables and are manipulated like any variable. To output a file variable you can just set it as any usual output:
val f = Val[File]

val producer = ScalaTask("""
  val f = newFile()
  f.content = "I love tofu"
""") set (
  outputs += f
)

Complementary information ๐Ÿ”—