Running multiple threads with Questa fails

[tasgomes]

I have an issue when running multiple threads using the Questa simulator and the flag -p. I created a minimal example to reproduce the issue.

The testbench implements a single test case:

library vunit_lib;
context vunit_lib.vunit_context;
context vunit_lib.vc_context;

entity test_tb is
    generic (
        runner_cfg : string
    );
end entity test_tb;

architecture tb of test_tb is
    constant TestRuntime_c : time := 1 us;
begin

    test_runner_watchdog(runner, 1000 ms);

    p_stimuli : process
    begin
        test_runner_setup(runner, runner_cfg);
        while test_suite loop

            if run("dummy_test") then
                wait for TestRuntime_c;
            end if;

        end loop;
        wait for 100 ns;
        test_runner_cleanup(runner);
    end process;

end architecture tb;

In the run.py, I replicated the test case 10 times:

    # Generate multiple test cases
    tb = lib.test_bench("test_tb")
    for test in tb.get_tests():
        for t in range(10):
            test.add_config(name="Test%d" % (t))

It is expected that all test cases are executed and pass. This is true when single-thread is used. However, when running in multi-thread, sometimes there are test cases that fail due to some "lock" mechanism:

Trimmed log:

$ python -u run.py -o vunit_out/questa -p 8 --simulator questa --simulator-path C:/intelFPGA_pro/21.2/questa_fe/win64
(...)
pass (P=6 S=0 F=0 T=10) lib_test_tb.test_tb.Test6.dummy_test (11.4 seconds)

pass (P=7 S=0 F=0 T=10) lib_test_tb.test_tb.Test2.dummy_test (13.4 seconds)

pass (P=8 S=0 F=0 T=10) lib_test_tb.test_tb.Test3.dummy_test (15.4 seconds)

# vsim -modelsimini C:/temp/questa_bug/sim/vunit_out/questa/modelsim/modelsim.ini -wlf C:/temp/questa_bug/sim/vunit_out/questa/test_output/lib_test_tb.test_tb.Test5.dummy_test_a507c5d1d48e12c839888281ffbf95fb8c4792c1/modelsim/vsim.wlf -quiet -t 1ps -onfinish stop lib_test_tb.test_tb(tb) -voptargs="+acc" -L vunit_lib -L osvvm -L lib_test_tb -g/test_tb/runner_cfg="active python runner : true,enabled_test_cases : dummy_test,output path : C::/temp/questa_bug/sim/vunit_out/questa/test_output/lib_test_tb.test_tb.Test5.dummy_test_a507c5d1d48e12c839888281ffbf95fb8c4792c1/,tb path : C::/temp/questa_bug/sim/tb/,use_color : true"
# Start time: 09:50:39 on Nov 24,2022
# ** Error: (vsim-1962) Can't create implicit optimized design in non-writable library C:/temp/questa_bug/sim/vunit_out/questa/modelsim/libraries/lib_test_tb.
# //  Questa Intel FPGA Edition-64
# //  Version 2021.2 win64 Apr 13 2021
# //
# //  Copyright 1991-2021 Mentor Graphics Corporation
# //  All Rights Reserved.
# //
# //  QuestaSim and its associated documentation contain trade
# //  secrets and commercial or financial information that are the property of
# //  Mentor Graphics Corporation and are privileged, confidential,
# //  and exempt from disclosure under the Freedom of Information Act,
# //  5 U.S.C. Section 552. Furthermore, this information
# //  is prohibited from disclosure under the Trade Secrets Act,
# //  18 U.S.C. Section 1905.
# //
# ** Warning: (vsim-6) -- Waiting for lock by "<user>@<computer_name>, pid = 21364
# ". Lockfile is "C:/temp/questa_bug/sim/vunit_out/questa/modelsim/libraries/lib_test_tb/_lock".
# ** Warning: (vsim-6) -- Waiting for lock by "<user>@<computer_name>, pid = 25780
# ". Lockfile is "C:/temp/questa_bug/sim/vunit_out/questa/modelsim/libraries/lib_test_tb/_lock".
# ** Warning: (vsim-6) -- Waiting for lock by "unknown (empty read)". Lockfile is "C:/temp/questa_bug/sim/vunit_out/questa/modelsim/libraries/lib_test_tb/_lock".
# Refreshing C:/temp/questa_bug/sim/vunit_out/questa/modelsim/libraries/lib_test_tb.test_tb(tb)
# ** Error (suppressible): (vcom-12110) All optimizations are disabled because the -novopt option is in effect. This will cause your simulation to run very slowly. If you are using this switch to preserve visibility for Debug or PLI features, please see the User's Manual section on Preserving Object Visibility with vopt. -novopt option is now deprecated and will be removed in future releases.
# ** Error: vcom failed.
# subinvocation of vcom failed; return status = 1.
# ** Error: (vsim-3171) Could not find machine code for 'C:/temp/questa_bug/sim/vunit_out/questa/modelsim/libraries/lib_test_tb.test_tb(tb)'.
# No such file or directory. (errno = ENOENT)
# Error loading design
Error loading design
# End time: 09:50:54 on Nov 24,2022, Elapsed time: 0:00:15
# Errors: 3, Warnings: 3
fail (P=8 S=0 F=1 T=10) lib_test_tb.test_tb.Test5.dummy_test (16.4 seconds)

pass (P=9 S=0 F=1 T=10) lib_test_tb.test_tb.Test1.dummy_test (18.5 seconds)

==== Summary ================================================
pass lib_test_tb.test_tb.Test4.dummy_test (3.3 seconds)
pass lib_test_tb.test_tb.Test8.dummy_test (1.8 seconds)
pass lib_test_tb.test_tb.Test7.dummy_test (6.4 seconds)
pass lib_test_tb.test_tb.Test9.dummy_test (2.9 seconds)
pass lib_test_tb.test_tb.Test0.dummy_test (9.3 seconds)
pass lib_test_tb.test_tb.Test6.dummy_test (11.4 seconds)
pass lib_test_tb.test_tb.Test2.dummy_test (13.4 seconds)
pass lib_test_tb.test_tb.Test3.dummy_test (15.4 seconds)
pass lib_test_tb.test_tb.Test1.dummy_test (18.5 seconds)
fail lib_test_tb.test_tb.Test5.dummy_test (16.4 seconds)
=============================================================
pass 9 of 10
fail 1 of 10
=============================================================
Total time was 98.8 seconds
Elapsed time was 18.5 seconds
=============================================================
Some failed!

As can be seen, the test cases do not communicate to each other so the goal of running in parallel is just to reduce execution time.
Multithreading using GHDL or Modelsim works fine.

The issue can be reproduced here:
questa_bug.zip

• Unzip
• Change directory to sim folder
• Execute command python -u run.py -o vunit_out/questa -p 8 --simulator questa --simulator-path C:/intelFPGA_pro/21.2/questa_fe/win64

Note that the issue is not reproducible 100%, sometimes it works (all test cases pass) but most of the times it does not.

Tools:
Python 3.9.0
VUnit 4.6.0
Questa Intel FPGA Edition-64 2021.2, Revision 2021.04, Date: Apr 13 2021

[LarsAsplund]

Do the -u option make any difference?

[tasgomes]

No, I have the same problem with or without it.

[LarsAsplund]

I've managed to gain access to a Questa licens. It's only one but if I run with many threads it generally works well. It simply waits for the license. I've seen the problem once though but I have not been able to repeat. I'm not sure what the "implicit optimized design" it tries to create is but a first test is maybe to turn optimizations off to see if the problem is avoided.

[eschmidscs]

I have tried to run Questa with several threads and licenses some time ago. I think this fails because questa always tries to write something to the compiled database. I did not manage to disable the optimization step included in vsim (it is basically an implicit vopt) so that it does not happen...
So one invocation of Questa kind of cripples the database for the next invocation...
I would appreciate a solution here, we might actually look into this as well given some time (not right now for me...).

[tasgomes]

Same here. I already fought against this optimization step... if you try to disable it then other issues pop up so we have to live with it.

Does VUnit provides an easy way to run the different test cases in different sub-folders inside the "vunit_out" folder?

[eschmidscs]

That would be a nice feature for running Questa in parallel: Tell vunit to build several vunit_out-folders with individual databases.
The really fancy version would then also merge the coverage from all these folders ;)

[harry-commin-enclustra]

I'm not sure if it is possible to completely disable Questa optimization. As far as I know, the best we can do is what @tasgomes already did in his example: pass -voptargs=+acc to vsim.

In my experience, this sidesteps a lot of bugs that crash Questa but not Modelsim. However, note that Questa is still printing this:

# ** Note: (vsim-3812) Design is being optimized...

So it seems like Questa is still doing something related to optimization, which Modelsim doesn't do.

I would be very interested in a solution/workaround for this issue.

[LarsAsplund]

There is no support today for multiple output directories. One could potentially create a workaround where a top-level run script calls a second level run script with different values for the --output-path option. That is not pretty so an integrated solution is to prefer. However, the important point is that with multiple directories there are multiple libraries to recompile. If you change something and run in 10 threads then there is a need to recompile 10 databases.

I think we would need support from Siemens to get a rationale for this behavior. No matter what solution you have for parallelization you would run into this problem. Do Siemens have their own solution for this? Anyone knows?

[LarsAsplund]

Assuming that it only needs write access when starting up the simulation one could have a solution that starts the individual threads with some temporal spacing. As a first test one could just set a fix time and then, if that works, one could handle it more dynamically by looking for lock files or similar. Could you try adding time.sleep(<some delay in seconds>) before this line:

vunit/vunit/test/runner.py

Line 414 in ce8db50

return self._tests[idx]

[harry-commin-enclustra]

@LarsAsplund That seems to work for me.

Without the time.sleep(), I get at least 1 error on roughly 50% of runs. With time.sleep(1), I only got 1 error out of > 10 consecutive runs, and it wasn't the same error as before:

** Error (suppressible): (vopt-19) Failed to access library 'work'

Here's the full printout from that test:

Starting lib_test_tb.test_tb.Test8.dummy_test
Output file: C:\temp\questa_bug\sim\vunit_out\test_output\lib_test_tb.test_tb.Test8.dummy_test_b0abc020e9342bcb19a5eeb3129e05a47ce4813c\output.txt
# vsim -modelsimini C:/temp/questa_bug/sim/vunit_out/modelsim/modelsim.ini -wlf C:/temp/questa_bug/sim/vunit_out/test_output/lib_test_tb.test_tb.Test7.dummy_test_87cd6de13db059111efe5554ba367c677b6b524f/modelsim/vsim.wlf -quiet -t 1ps -onfinish stop lib_test_tb.test_tb(tb) -voptargs="+acc" -L vunit_lib -L osvvm -L lib_test_tb -g/test_tb/runner_cfg="active python runner : true,enabled_test_cases : dummy_test,output path : C::/temp/questa_bug/sim/vunit_out/test_output/lib_test_tb.test_tb.Test7.dummy_test_87cd6de13db059111efe5554ba367c677b6b524f/,tb path : C::/temp/questa_bug/sim/tb/,use_color : true"
# Start time: 16:25:24 on Dec 01,2022
# ** Note: (vsim-3812) Design is being optimized...
# ** Error (suppressible): (vopt-19) Failed to access library 'work' at "C:/temp/questa_bug/sim/vunit_out/modelsim/libraries/lib_test_tb".
# ** Note: Vopt Compiler exiting
# ** Note: (vsim-12126) Error and warning message counts have been restored: Errors=1, Warnings=0.
# Error loading design
Error loading design
# End time: 16:25:24 on Dec 01,2022, Elapsed time: 0:00:00
# Errors: 1, Warnings: 0
fail (P=3 S=0 F=1 T=10) lib_test_tb.test_tb.Test7.dummy_test (1.7 seconds)

[LarsAsplund]

In order for us to refine this into a stable solutions we need to understand how Questa's locking mechanism works. Asking Siemens is a good start but I think that would require a support contract with them. Anyone that has that?

[tasgomes]

I found the following in the Questa SIM User's Manual:

If you have a Questa installation, you can find this document in menu Help -> Questa Sim PDF Bookcase. This will open a PDF with links to several other documents, one of them is the Questa SIM User's Manual.

[LarsAsplund]

I think the fundamental problem here is that vopt is run implicitly as part of the vsim call. This means that VUnit has no good control of the lock files. If two simulation threads are started at the same time they might try to run vopt at the same time and there is problem. If their startup times differ their vopt runs may not collide but a vopt run can also collide with that of a thread that started earlier. A more controlled approach would be to have a three-step flow where vopt is run explicitly between compilation and simulation. VUnit won't start any simulation threads before the design is optimized just as it won't start simulation before compilation is completed. This is assuming that vsim doesn't create a lock when figuring out if vopt was already run or not. This is more work though so I would like to create a workaround that can also work as a proof of concept. To be a usable workaround it must not modify any VUnit code the way we did in the first attempt. I'm thinking a function hook running before the tests. The first test to call the hook will run vopt for the whole design and all other theads calling that hook will simply wait for that to complete.

[LarsAsplund]

Try adding this class to your run script

from threading import Lock
from vunit.ostools import Process


class VOpt:
    def __init__(self, vunit):
        self._lock = Lock()
        self._optimization_complete = False
        self._vunit = vunit
        for lib in vunit.get_libraries():
            for tb in lib.get_test_benches(allow_empty=True):
                tb.set_pre_config(self)

    def __call__(self, output_path):
        self._lock.acquire()

        if not self._optimization_complete:
            for lib in self._vunit.get_libraries():
                for tb in lib.get_test_benches(allow_empty=True):
                    try:
                        args = [
                            str(Path(self._vunit._simulator_class.find_prefix()) / "vopt"),
                            "-work",
                            lib.name,
                            tb.name,
                            "-o",
                            f"{tb.name}_opt",
                        ]

                        Process(args, cwd=str(Path(self._vunit._args.output_path) / "modelsim" / "libraries"))
                    except Process.NonZeroExitCode:
                        return False

            self._optimization_complete = True

        self._lock.release()

        return True

Then just before you call the main method

VOpt(prj)
prj.main()

The optimized design must be stored under a different name so change this line:

vunit/vunit/sim_if/modelsim.py

Line 269 in ce8db50

config.library_name + "." + config.entity_name + architecture_suffix,

to

config.library_name + "." + config.entity_name + "_opt" + architecture_suffix,

I only have one license to test with but the note about vsim optimizing the design has disappeared so from that point of view it looks good. vopt is run before vsim and it is run from a single thread.

[tasgomes]

@LarsAsplund

I added the class to my sim/run.py script. And before I call main method, I call this class. I made a small change since my VUnit object is called vu:

    # Call VUnit main
    VOpt(vu)
    vu.main(post_run=common.post_run)

Then I modified the vunit/vunit/sim_if/modelsim.py as described.

Now, I get this error even if I run in single-thread mode, so either I did something wrong or something else is still missing:

$ python -u run.py -o vunit_out/questa --simulator questa --simulator-path C:/intelFPGA_pro/21.2/questa_fe/win64
WARNING - C:\temp\questa_bug\lib\vunit\vunit\vhdl\osvvm\ScoreboardPkg_int_c.vhd: package 'scoreboardpkg_int' previously defined in C:\temp\questa_bug\lib\vunit\vunit\vhdl\osvvm\ScoreboardPkg_int.vhd
WARNING - C:\temp\questa_bug\lib\vunit\vunit\vhdl\osvvm\ScoreboardPkg_slv_c.vhd: package 'scoreboardpkg_slv' previously defined in C:\temp\questa_bug\lib\vunit\vunit\vhdl\osvvm\ScoreboardPkg_slv.vhd
Re-compile not needed

Starting lib_test_tb.test_tb.Test0.dummy_test
Output file: C:\temp\questa_bug\sim\vunit_out\questa\test_output\lib_test_tb.test_tb.Test0.dummy_test_b066f085a2ef9354b56fc80f6ed1a5f7aee994eb\output.txt
# vsim -modelsimini C:/temp/questa_bug/sim/vunit_out/questa/modelsim/modelsim.ini -wlf C:/temp/questa_bug/sim/vunit_out/questa/test_output/lib_test_tb.test_tb.Test0.dummy_test_b066f085a2ef9354b56fc80f6ed1a5f7aee994eb/modelsim/vsim.wlf -quiet -t 1ps -onfinish stop lib_test_tb.test_tb_opt(tb) -voptargs="+acc" -L vunit_lib -L osvvm -L lib_test_tb -g/test_tb/runner_cfg="active python runner : true,enabled_test_cases : dummy_test,output path : C::/temp/questa_bug/sim/vunit_out/questa/test_output/lib_test_tb.test_tb.Test0.dummy_test_b066f085a2ef9354b56fc80f6ed1a5f7aee994eb/,tb path : C::/temp/questa_bug/sim/tb/,use_color : true"
# Start time: 10:04:01 on Dec 07,2022
# ** Note: (vsim-3812) Design is being optimized...
# ** Error: (vopt-13131) Failed to find design unit test_tb_opt(tb).
#         Searched libraries:
#             vunit_lib
#             osvvm
#             lib_test_tb
#             work
# Optimization failed
# ** Note: (vsim-12126) Error and warning message counts have been restored: Errors=1, Warnings=0.
# Error loading design
Error loading design
# End time: 10:04:01 on Dec 07,2022, Elapsed time: 0:00:00
# Errors: 1, Warnings: 0
fail (P=0 S=0 F=1 T=10) lib_test_tb.test_tb.Test0.dummy_test (1.4 seconds)

[LarsAsplund]

I see that there is a questa in your VUnit output path (C:\temp\questa_bug\sim\vunit_out\questa\test_output). Did you change the output path from the default on the command line? vopt must run in the correct directory and should take any command line options:

Process(args, cwd=str(Path(self._vunit._args.output_path) / "modelsim" / "libraries"))

I would print str(Path(self._vunit._args.output_path) / "modelsim" / "libraries") to see where that goes. You should see vopt generating a directory called test_tb_opt. You can also change the Python line above to this to see that vopt is active and is doing something.

proc = Process(args, cwd=str(Path(self._vunit._args.output_path) / "modelsim" / "libraries"))
proc.consume_output()

[tasgomes]

Yes, I changed the output path from the default on the command line. You can see it in the first line of the previous console output:

Since sometimes I run simulation using Modelsim or Questa, I like to have separated folders. This also allows me to run the same testbench in different simulators at the same time.

I guess your workaround should work with this option then.

To avoid this issue for now, I removed the -o parameter from the command line and I still have the same issue:

$ python -u run.py --simulator questa --simulator-path C:/intelFPGA_pro/21.2/questa_fe/win64
WARNING - C:\temp\questa_bug\lib\vunit\vunit\vhdl\osvvm\ScoreboardPkg_int_c.vhd: package 'scoreboardpkg_int' previously defined in C:\temp\questa_bug\lib\vunit\vunit\vhdl\osvvm\ScoreboardPkg_int.vhd
WARNING - C:\temp\questa_bug\lib\vunit\vunit\vhdl\osvvm\ScoreboardPkg_slv_c.vhd: package 'scoreboardpkg_slv' previously defined in C:\temp\questa_bug\lib\vunit\vunit\vhdl\osvvm\ScoreboardPkg_slv.vhd
Re-compile not needed

Starting lib_test_tb.test_tb.Test0.dummy_test
Output file: C:\temp\questa_bug\sim\vunit_out\test_output\lib_test_tb.test_tb.Test0.dummy_test_b066f085a2ef9354b56fc80f6ed1a5f7aee994eb\output.txt
# vsim -modelsimini C:/temp/questa_bug/sim/vunit_out/modelsim/modelsim.ini -wlf C:/temp/questa_bug/sim/vunit_out/test_output/lib_test_tb.test_tb.Test0.dummy_test_b066f085a2ef9354b56fc80f6ed1a5f7aee994eb/modelsim/vsim.wlf -quiet -t 1ps -onfinish stop lib_test_tb.test_tb_opt(tb) -voptargs="+acc" -L vunit_lib -L osvvm -L lib_test_tb -g/test_tb/runner_cfg="active python runner : true,enabled_test_cases : dummy_test,output path : C::/temp/questa_bug/sim/vunit_out/test_output/lib_test_tb.test_tb.Test0.dummy_test_b066f085a2ef9354b56fc80f6ed1a5f7aee994eb/,tb path : C::/temp/questa_bug/sim/tb/,use_color : true"
# Start time: 15:30:43 on Dec 07,2022
# ** Note: (vsim-3812) Design is being optimized...
# ** Error: (vopt-13131) Failed to find design unit test_tb_opt(tb).
#         Searched libraries:
#             vunit_lib
#             osvvm
#             lib_test_tb
#             work
# Optimization failed
# ** Note: (vsim-12126) Error and warning message counts have been restored: Errors=1, Warnings=0.
# Error loading design
Error loading design
# End time: 15:30:43 on Dec 07,2022, Elapsed time: 0:00:00
# Errors: 1, Warnings: 0
fail (P=0 S=0 F=1 T=10) lib_test_tb.test_tb.Test0.dummy_test (1.5 seconds)

I already realized I was missing from pathlib import Path since you are now adding Path.

I don't see any test_tb_opt directory:

Maybe the VOpt object is not even running??

[LarsAsplund]

The optimized design should be a directory in lib_test_tb. Something in there? If you added the consume_output() method call and run with -v you should see vopt progress reporting like optimizing package xyz

[tasgomes]

Oh I think I got it. I have a master run.py that gathers all the run.py present in my project. I was adding the code from here in the wrong run.py file.

I get now this one:

$ python -u run.py --simulator questa --simulator-path C:/intelFPGA_pro/21.2/questa_fe/win64 *Test0* -l
WARNING - C:\temp\questa_bug\lib\vunit\vunit\vhdl\osvvm\ScoreboardPkg_int_c.vhd: package 'scoreboardpkg_int' previously defined in C:\temp\questa_bug\lib\vunit\vunit\vhdl\osvvm\ScoreboardPkg_int.vhd
WARNING - C:\temp\questa_bug\lib\vunit\vunit\vhdl\osvvm\ScoreboardPkg_slv_c.vhd: package 'scoreboardpkg_slv' previously defined in C:\temp\questa_bug\lib\vunit\vunit\vhdl\osvvm\ScoreboardPkg_slv.vhd
Traceback (most recent call last):
  File "C:\temp\questa_bug\sim\run.py", line 68, in <module>
    VOpt(vu)
  File "C:\temp\questa_bug\sim\run.py", line 19, in __init__
    for lib in vunit.get_libraries():
AttributeError: 'VUnit' object has no attribute 'get_libraries'

[tasgomes]

Maybe it is related with how I load the VUnit object?

[LarsAsplund]

get_libraries is fairly new so it might not be included unless you've cloned from GitHub. Update or replace that call with a list containing the library objects for the libraries containing tests. [vunit.library("lib_test_tb")] should be sufficient in your case.

[tasgomes]

@LarsAsplund I updated the VUnit repo from 4.6.0 to master and get_libraries is not a problem anymore.

After that, the parallel execution works fine with Questa :)

Do you think the work-around below can make it to next release?

[LarsAsplund]

Adding a native three-step approach is a bit more work. Since we haven't released for more than a year I feel that we should take what we have (more or less) and make a release of that first before committing to more work. If we can think of a nice API to avoid modifying modelsim.py we may be able to squeeze that in as a temporary solution.

[tasgomes]

@LarsAsplund Do you think now would be a good time to introduce the fix to allow Questa running with multiple threads?

[LarsAsplund]

@tasgomes Going forward I'm thinking about theme-based releases where we gather related issues. I've yet to decide on themes and order of execution but I'm pretty sure that there would be a Questa theme and an OSVVM theme if we go with this approach

[harry-commin-enclustra]

I had a lot of trouble reproducing this fix, so I have attached an updated zip file with the fix included.

questa_bug.zip

My changes were:

• Updated VUnit to the latest commit (7d7ae16).
  • Added vu.add_vhdl_builtins() as described in #777.
• Removed the if __name__ == '__main__': part from sim/tb/run.py. I could be wrong, but I don't think it makes sense to call that script directly, so it was just creating confusion about where to use VOpt.
• Modified sim/run.py and lib/vunit/vunit/sim_if/modelsim.py as described in the posts above.

The part that I really got stuck on was that this change is required (it's not just for printing out debug info):

proc = Process(args, cwd=str(Path(self._vunit._args.output_path) / "modelsim" / "libraries"))
proc.consume_output()

[LarsAsplund]

This is solved with #899