Ares

Sadly, due to the way classes are loaded and the class path works when testing student code with maven, there are still vulnerabilities if students manage to load classes that would be in trusted packages. This is especially problematic if they shadow library classes, such as JUnit’s Assertions.

To prevent that, you have to use the Maven Enforcer Plugin to make sure no student content lands in trusted packages:

Maven:

Gradle:

You can use @WhitelistPath and @BlacklistPath to control access to paths. By default, no access is granted, and so you need to use @WhitelistPath to give student code the permission to read and write files explicitly. You can specify exceptions using @BlacklistPath which will overpower the whitelisted paths.

The following examples will make use of course1920xyz as placeholder value for the real Artemis exercise name/id. Replace it with the real one when borrowing code snippets, or nothing will work as expected.

Most importantly, this does not only apply to explicit file IO, but also to the .class files that the class loader reads, as needed. This already happens if one student class requires another one, that has not been loaded after that. You can recognize that in the standard error output:

This usually means the class loader could not load the class. The parentheses show, that the problem is the missing whitelisting. Therefore, all test setups should have some whitelisting.

A number of examples how you can whitelist paths in Ares:

That was not everything but already quite a lot. Take a look at the Javadoc of the annotations and enums used, if you want to know more. Before you give up, here is my recommendation how to start:

Add a @BlacklistPath for other important classes, like your reference implementations of the solution to test against should you use that approach.

Note: the Artemis project starts with course1920xyz, but the build in Bamboo (by Artemis) will happen in a directory named after the build plan, which is in upper case and therefore, begins with COURSE1920XYZ. Make sure that you do not build multiple student solutions in the same directory on the same machine using the git clone (lower case) approach. Otherwise, adjust the whitelisting to your needs.

JUnit already provides means of applying timeouts to tests. However, those are not strict in the sense of "enforced in the strongest possible way". What is meant by that?

There are three different ways how the timeouts can work:

Rule 1: When testing with Ares, always use @StrictTimeout for timeouts, the others will not work reliably, especially in conjunction with the Ares security.

Rule 2: When writing tests for Artemis, always use @StrictTimeout. There is no reason to omit the timeout, since you do not know the code students will write. (And they will write code spawning millions of threads in endless loops, which in turn will do the same recursively.)

By default, Ares will record standard and error output of each test internally and not print it to the console. The recorded output can then be obtained and tested, see IOTester The reason for this is on the one hand to keep the console and logs short and clean and on the other hand prevent students from accidentally messing up the logs with millions of lines. Ares also has a hard limit on the total number of printed chars at around 10 million.

To mirror the output recorded by Ares to the console, use the @MirrorOutput annotation on the test class or method.

It is also worth noting that Ares enforces valid UTF-8 being printed and throws an appropriate exception otherwise.

Hidden tests will be executed by Ares only after the deadline. This poses the problem, how the exercise creators should work on the tasks, tests and the sample solution. One possible solution would be to use an alternative deadline annotation or change the deadline temporarily. The problem is that it is quite likely one might forget to change it back again, and protecting the hidden tests would fail.

Use @ActivateHiddenBefore just like @Deadline to state the LocalDateTime before which hidden tests should be executed. This date should, of course, be before the release of the exercise on Artemis.

You can use @ExtendedDeadline together with a duration like 1d or 2d 12h 30m to extend the deadline by the given amount. @ExtendedDeadline("1d"), for example, extends the deadline by one day. If you use the annotation on different levels (e.g. class and method) without stating a new deadline (e.g. deadline only on class level), the extensions will be added together.

By default, Ares will not allow non-whitelisted code to use threads at all. That includes thread pools, but excludes the common pool and its users, like parallel streams. To allow the use of Threads, use the annotation @AllowThreads. The number of active threads is also limited, the default value of that is 1000, but can be changed in the annotation. Please keep in mind that this limit should not be larger than 1000 to prevent performance and timeout chaos.

New threads are for security reasons not directly whitelisted by Ares and will not be allowed to do anything security critical. If you trust a thread (at least its entry point), you can explicitly request the thread to be whitelisted using ArtemisSecurityManager.requestThreadWhitelisting(Thread). The thread calling the method and its stack must be whitelisted, of course.

One example showing some possibilities here:

Note that Ares normalizes the line breaks to \n, and OutputTester offers many different approaches to checking output (e.g. single string, list of strings, …​).

If students read more lines than provided, they get the following feedback: …​ java.lang.IllegalStateException: no further console input request after the last(number 1: "5") expected. …​

See also IOTester and for more examples, the InputOutputUser test.

Showing Standard Output covers how the student output is managed and shown in the test logs.

Ares allows for local network connections by using the @AllowLocalPort annotation.

There are plenty of configuration options, and the code can get complicated quickly due to the threads required to test network connections. One issue can be that waiting network connections block threads in such a way that they cannot be stopped (waiting in native code), so we recommend using timeouts for connections at least on one end consistently.

For examples, have a look at the test NetworkUser.

You can set a locale for Ares (and the rest of Java) by adding the @UseLocale JUnit extension to classes/methods, which will set the Java default locale that is also used by Ares. The locale is changed only for the scope where the annotation is applied.

Ares is currently localized in German (de_DE) and English (en_US), where en_US is the fallback for any other locale.

See also the LocaleUser test for more examples.