Async tests (wait) sometimes fail

[finestructure]

I've got a test set up to wait for cache to be filled based on requests being sent (mocked via OHHTTPStubs).

- (void)test_fetchAdsIfNeeded
{
    [self mockResponseWithBlock:^OHHTTPStubsResponse *{
        return [self uniqueAdResponse];
    }];

    AdCache *cache = [AdCache currentValue];
    expect(cache.ads).to.haveCountOf(0);
    [cache fetchAdsIfNeeded];
    expect(cache.ads).will.haveCountOf(10);

    [OHHTTPStubs removeLastRequestHandler];
}

Sometimes this test will fail. It's not deterministic. My suspicion is that it is related to runloop event processing.

Now I'm far from an expert on this but I've compared applyMatcher: to my own async test extension which is based on GHUnit [1] and noticed that there's additional runloop processing going on in GHUnit.

I'll submit a pull request for this, which in my testing fixed the issue (but as non-deterministic issues go, this might not be the end-all fix).

[1] http://feinstruktur.com/blog/2012/8/12/unit-testing-asynchronous-code.html

[finestructure]

I've just had it happen again, so this change does not fix the problem. Needs more investigation as to what the differences are between the two approaches. My SenTestKit category definitely doesn't show this behaviour, it only started showing up when converting one of our tests to expecta format. I much prefer expecta's syntax and conciseness - will try and get this to work.

[dpassage]

@sas71, any progress? I've started seeing this as well, although it could be related to upgrading to Xcode 5.

I've tried 3 or 4 different Obj-C unit testing solutions, looking for something that can deal with asynchronicity in a robust fashion, and have yet to find it.

[finestructure]

Not yet, unfortunately. Have you tried the version I've linked to above? Just to get an idea if there's a working version to compare to or if there's a general problem.

[petejkim]

could you give specta's AsyncBlock feature a try?

[finestructure]

I've had some time to revisit this and just noticed that these failures only seem to happen when using Xcode 5. I'll try to dig deeper. I'll also look at AsyncBlock as soon as I find the time, Peter.

[brennon]

Any progress? I have the same problems when I rig my test using an AsyncBlock from Specta.

[finestructure]

I’m afraid not. We’ve had to abandon this for now due to lack of time to investigate further.

Dr. Sven A. Schmidt
[email protected] | http://feinstruktur.com

On 7 Jan 2014, at 17:56, Brennon Bortz [email protected] wrote:

Any progress? I have the same problems when I rig my test using an AsyncBlock from Specta.

—
Reply to this email directly or view it on GitHub.

[tonyarnold]

FWIW, it appears that there may have been an underlying bug in Xcode. If you have some time, give the same tests another go under Xcode 5.1 — I'm seeing much more consistent results now.

[tonyarnold]

Nope, I'm still seeing this in Xcode 6.0.1. Something is not right, but I haven't been able to figure out what.

@petejkim what are the differences in how Specta's AsyncBlock handles this compared to Expecta's will/willNot?

[fatuhoku]

will/willNot is not reliable for me at all. It works for a while — long enough for me to believe that a test has been fixed — but when I implement a few more tests, the tests involving will start hanging indefinitely, preventing normal test execution from occurring.

If it hangs once, it consistently hangs.

Pretty mysterious. I've not set the timeout value at all either.

UPDATE: curiously still, I tried using https://github.com/hfossli/AGAsyncTestHelper to replace the will calls, but I get the nonterminating behaviour.

Could it be because I'm trying to check against a Core Data attribute that's causing the loop?

With Expecta:

expect(recipe.needsSync).will.equal(@YES);

With AGAsyncTestHelper:

AGWW_WAIT_WHILE_NOT_EQUALS(recipe.needsSync, @YES, 0.5);

[polqf]

Is there any update about it? I am sometimes getting random fails when using will

Also, it sometimes get stuck when applyMatcher:to gets called. Here:

Line 123

 [[NSRunLoop currentRunLoop] runUntilDate:[NSDate dateWithTimeIntervalSinceNow:0.01]];

[rafaelnobrepd]

I started using Expecta very recently and was very happy and impressed, but I'm having some nondeterministic trouble using will matching as well, in my case it only happens for a specific test running on Travis. It passes on Xcode and fastlane locally, but fails on CI.