Tensorflow Tf.group Ignoring Dependencies?

Following on from an earlier question, it seems tf.group is indeed ignoring dependencies. Here's a simple stand-alone example (I have run it on Python 2.7 with TensorFlow 1.1): imp

Solution 1:

Using tf.control_dependencies correctly does solve this problem:

import tensorflow as tf

xs = [tf.constant(x) for x inrange(10)]
dependency = None
dxs = []

for x in xs:
    if dependency isNone:
        dependency = tf.Print(x, [x])
    else:
        with tf.control_dependencies([dependency]):
            dependency = tf.Print(x, [x])

    dxs.append(dependency)

print_all_op = tf.group(*dxs)

with tf.Session() as session:
    session.run(print_all_op)

Note that the Print operation needs to be created within the tf.control_dependencies context manager.

I am still unclear why the control_flow_ops.with_dependencies version fails.

Solution 2:

I think the problem is that the initial code creates dependencies between dummy identity ops implicitly created by control_flow_ops.with_dependencies and not the actual tf.Print ops. Tensorflow seems to only ensure that the ops in the dependency list have been already executed but the order of other preceding ops is not fixed. In the above example, the dependencies are created on the dummy identity ops created by control_flow_ops.with_dependencies:

dependency = control_flow_ops.with_dependencies([dependency], x)

which should be equivalent to:

        with tf.control_dependencies([dependency]):
            dependency = tf.identity(x)

Thus, the dependencies here are created between the tf.identity ops and not the tf.Print ops. The tf.Print ops can be run in any order, the strict ordering is only on the tf.identity ops. I don't think it is possible to achieve the desired behavior with control_flow_ops.with_dependencies. Instead one has to use with tf.control_dependencies instead (as already suggested by the op):

xs = [tf.constant(x) for x inrange(10)]
dependency = None
dxs = []

for x in xs:
    if dependency isNone:
        dependency = tf.Print(x, [x])
    else:
        with tf.control_dependencies([dependency]):
            dependency = tf.Print(x, [x])

    dxs.append(dependency)