Improve error messaging for the katas that allow it

[tcNickolas]

Several katas could have better error messaging:

• Superposition : the test harness could log the actual state of the system after prep vs the expected state before asserting that they are the same (the change would add state prep and logging using DumpMachine to AssertEqualOnZeroState)
• BasicGates : a similar improvement, but a bit more extra code to write a unified test wrapper (since all tests in it just use AssertOperationsEqualReferenced). Since the actual test compares the unitaries, we might want to add a DumpDiffOnOneInput that would:
  • prepare some nice state like 0.8 |0⟩ + 0.6 |1⟩,
  • apply the reference solution on it,
  • dump the state to get expected state,
  • repeat the sequence for the user's solution.
  • the overall output should look similar to the style used in Superposition kata (as done in Improve error messaging for Superposition. #214)
• Measurements : the test harness already reports the number of incorrect classifications; in addition, it can print the states which were misclassified in human-readable format, taking an array of string representations of states as an extra parameter to operations like DistinguishTwoStates_OneQubit and logging the mismatches. We'd have to be careful not to log 50 lines of mismatches, though... Maybe keep track of different kinds of mismatches and report aggregates in the end of the test (e.g., "Classified |0⟩ as |1⟩ 15 times, classified |1⟩ as |0⟩ 21 times")?
• JointMeasurements : same as in Measurements
• DeutschJozsaAlgorithm
• SolveSATWithGrover
• and GraphColoring : use logging approach similar to RippleCarryAdder to produce more readable results for testing oracles on basis states (so that the solver knows which input state is processed incorrectly, not just that there is an error).

There are probably other katas that allow similar improvements - please share the ideas in the comments!

[sandreenko]

I will work on the first task (Superposition).

[ulitoo]

I will be working on the Measurements test output improvement

[vivanwin]

I will work on the BasicGates, to log the actual state.

[jainvasu631]

@tcNickolas What kind of messaging did you have in mind for SolveSATWithGrover?

[tcNickolas]

@jainvasu631 Apologies for the delay replying!

For the oracle-based tasks, there are two ways to verify that they are done correctly:

• One can compare the solution to the reference implementation using AssertOperationsEqualReferenced - this is faster but the result it gives is "incorrect qubit state", which is not particularly helpful.
• Or one can iterate through all possible inputs, convert them to basis states, prepare them, apply the oracle to them and measure the results - if the classical result is incorrect (or if the basis states measured changed as a result of applying the oracle), the implementation is incorrect, but you know what's the basis state you need to debug.

Currently most of the katas use a mixed approach - the basis check to make sure the solution behaves correctly on the basis states (in a "classical" case), and the unitary comparison to make sure that there are no unexpected phases creeping in somewhere. The error messaging would be helpful for the basis check part, telling the learner "Your solution returns 0 for input 101 when it should return 1" or "Your solution modified the state of input qubits for input 101 (it became 110)".

[jainvasu631]

Since I worked on the SolveSATWithGrover Workbook, I remember that the testing harness is incredibly slow for certain tasks. Specifically it takes significant time for Tasks 1.5, 1.6 and Task 2.2. When I investigated the slowdown I saw that they include the check AssertOracleImplementsFunction which essentially does what you describe (iterating through basis states and checking the answer), we just need to add a message statement when the assert fails.

I think I can work on this.

Also since DeutschJozsaAlgorithm is also Oracle based, I can try to do the same and implement the AssertOracleImplementsFunction with the improved checking and error statements.

[tcNickolas]

@jainvasu631
Sounds good - I would start with one of them (probably SolveSATWithGrover) so that we can polish our approach, and then carry it over to the other two once it's polished.

You make a good point about the tests as well. I spent a bit of time looking into why are they so slow when I was reviewing your workbook PR, and opened #507 for refactoring them - we can make them faster without modifying the way the testing is done, just by being more careful about the test instances we're using. I'm hoping somebody will pick it up in the next couple of weeks, since it's a pretty good first issue for somebody to do - if not, and if you're willing, I'll welcome your help :-)